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Foreword 



For the first time, quality assurance experts and expertise were the focus of an ENQA 
workshop, which took place in Oslo in February 2008. 

External reviews, performed by academic peers, and other reviewers such as 
students, QA-experts and stakeholders, and coordinated by quality assurance agencies, 
have become the preferred method of national quality control of higher education 
institutions and/or programmes in most signatory countries of the Bologna Process. 

The workshop demonstrated that there is a need to reflect on the experiences and 
knowledge acquired by agencies and higher education institutions. 

The present report focuses on the people behind the review activity, namely the 
experts and agency staff. The experts’ judgements have sometimes a significant impact 
on individuals, programmes and institutions. Hence, the role of the experts and the 
balance between expertise and regulated procedures and standards are fundamental 
questions. Expert knowledge, in the form of independent peer assessment, remains 
absolutely crucial for the legitimacy and relevance of external quality assurance 
of higher education. The report also analyses the peer review model itself, which 
still needs to be further examined by the academic communities concerning to the 
questions the method can and should address. 

Finally, the review reports produced by the experts need to be placed in a context, 
and this clearly merits a wider discussion of its own. The general public, the reviewed 
institution and all its stakeholders, the government, the quality assurance agencies 
and the expert panel itself are involved and affected by these reports. Therefore it is 
important to consider how the results of peer reviews are presented. 

I hope this report will provide ENQA members and other stakeholders with inspiring 
insights in order to take the debate forward. 



Bruno Curvale, 

President 

European Association for Quality Assurance in Higher Education (ENQA) 



Chapter 1: Introduction 

1.1 Introduction to the report 

This report is a product of an ENQA Workshop “Assessing educational quality: 
Knowledge production and the role of experts” hosted by the Norwegian Agency for 
Quality Assurance in Education (NOKUT) in Oslo in February 2008. On that occasion 
the ENQA pool of experts, nominated by ENQA members and key European partner 
organisations, was invited to an ENQA event. The workshop gathered representatives 
from higher education institutions and agencies to exchange and discuss experiences on 
the process of external reviews and on the role of the review experts in that process. 

The workshop and this publication flourished from the “Expert knowledge” 
project carried out by NOKUT. The project focused on the status and legitimacy of 
the knowledge acquired through external assessments of higher education, in which 
the role of external reviewers (experts) as well as the cooperation between experts 
and quality assurance agencies are crucial. This report consists of articles that were 
submitted by workshop presenters and participants, upon request from the ENQA 
Secretariat. Articles by Professor em. Berit Askling from the University of Gothenburg 
and Professor Kirsten Hofgaard Lycke from the University of Oslo, as well as the article 
by Gro Hanne Aas and Wenche Froestad from NOKUT reflect on the results from the 
above-mentioned project. 

In his article, Professor Peder Haug from the Volda University College investigates 
the potential of the ‘expert panel methodology’ in evaluations and compares it to 
’research-based evaluations’. The article by Anne Karine Sorskar from NOKUT 
discusses the results from a questionnaire that was sent to the participating agencies 
prior to the workshop concerning the agencies’ use of experts in external reviews. 

In his article, Karl Dittrich, the Chairman of the Accreditation Organisation of the 
Netherlands and Flanders (NVAO), examines the practices in the publication of review 
reports in different countries also reflecting on the advantages and disadvantages of 
publishing these reports. Sirpa Moitus and Riitta Pyykko from the Finnish Higher 
Education Evaluation Council (FINHEEC) contribute to the report by presenting ways 
of promoting the usefulness and utilisation of evaluations. 

The workshop, and thus this publication, emphasise the importance of feedback and 
discussions with stakeholders on the evaluations, and in most cases these evaluations 
are carried out by expert panels. 

1.2 Introduction to topics of the workshop 

The overall theme of the workshop was the knowledge production by external 
panels assessing higher education and the role that the expert panels play in 
external quality assessments. To elaborate this rather generic theme, the workshop 
focused on three more distinct topics: 



1 The project involved four experts from the field of educational research; Berit Askling, Jon Frode Blichfeldt, Kirsten Hofgaard 
Lycke and Svein Mot he, as well as three advisers from the staff of NOKUT; Gro Hanne Aas, Wenche Froestad and Ole Espen 
Rakkestad. See also the paper “Quality Tellers: The status of knowledge produced by expert panels assessing educational quality” 
(available at www.nokut.no). 



• External expert panels’ evaluations vs. research-based evaluations of higher 
education 

• “The quality contract”: Agency regulations meet expert panels 

• Expert knowledge in public 

External expert panels' evaluations vs. research-based evaluations of higher 
education 

Governance and control of higher education through national quality regimes vary, but 
quality assessments by external expert panels play a crucial role in most countries. 

A remaining topic for discussion, however, is the respective merits of expert panels’ 
evaluations versus more research based evaluations. The debates too often take the form 
of stalled discussions between (educational) researchers and, for example, representatives 
of quality assurance agencies, where the defence of their own position easily comes to the 
fore. The workshop aims at fresh, reflective, informed and developmental discussions on 
this controversial issue. (Workshop announcement) 

The first presentation of the workshop by Professor Peder Haug compared the external 
expert panels’ evaluations and research-based evaluations of higher education. Haug 
stated that: “The expert panel approach is most reliable when it comes to questions 
concerning the structural elements of a programme or of an institution, for instance the 
number of staff, their competence and grades; the number of students, accomplishments, 
programme organisation, systems for control of quality, etc. To answer process and result 
questions, other models of evaluation could be more appropriate, for instance research- 
based evaluation.” 

In the discussion following the presentation, it was commonly agreed that for some 
types of evaluations, complementing approaches where QA agencies cooperate with 
research institutions, should be open-mindedly considered. The objective and purpose 
of the assessment exercise are the decisive factors when choosing the evaluation 
method. 

The outcomes of assessments may provide a basis for decisions about organisational 
change, funding, or similar. The workshop participants discussed whether such 
decisions could be based on the expert panels’ evaluations, or if a research-based 
evaluation would produce more valid and reliable data. 

“The quality contract": Agency regulations meet expert panels 

Experts assessing higher education on behalf of QA agencies work within a national model 
of external quality assurance and get their mandates from the QA agencies. Evaluation 
procedures as well as standards and criteria are to a large extent regulated by agencies 
or authorities. Such regulations are important for the process to be transparent and 
predictable. In addition, they ensure that the evaluated institutions are assessed according 
to the same requirements and criteria. However, experts being responsible for conducting 
the assessments, including the pronounced evaluations, raise the question of how much 
room there is for “proper” expert knowledge. The focus will be on how agencies and 
experts handle this balance. (Workshop announcement) 



To start the discussions in this session, two external experts of NOKUT, Karen Junker, 
the Director of education at County Governor in Aust-Agder, and Kirsten Hofgaard 
Lycke, Professor at the University of Oslo, presented some of their experiences on 
working in expert panels. Christian Thune, former President of ENQA, presented the 
needs for regulation as seen from the view of quality assurance agencies. 

Building on the previous debate on evaluation methods, the participants discussed 
the significance of trust. In general, the overall experience of the agencies is that 
external experts perform well in review panels. Still, regulations have become more 
detailed, especially in the area of accreditation and where avoiding conflicts of 
interpretation is particularly important. The importance of trust was pointed out in the 
workshop. To obtain successful assessments, there must be trust between the agency 
and the experts, as well as between the agency, experts and the evaluated institutions. 

A crucial issue, in theory and practice, is the selection of experts by quality 
assurance agencies. The participants were, prior to the workshop, asked to answer a 
few questions on the tasks and the status of experts in their own agencies. Advisor 
Anne Karine Sorskar from NOKUT presented the summary of the results. The choice 
of experts varies depending on the type of evaluation. As a current practice, experts 
are often recruited from the following groups: academic peers, students, QA experts 
and stakeholders. The training of experts and the agency regulations that concern their 
tasks are crucial issues. 

The European Consortium for Accreditation in higher education (ECA) has 
developed principles for the selection of experts listing competencies that panel 
members should have for conducting institutional and programme accreditation 
assessment. These principles were presented at the workshop by Rolf Heusser, the 
Director of Center for Accreditation and Quality Assurance of the Swiss Universities 
(OAQ). In the discussions it was apparent that informal aspects, such as reputation and 
social skills, have an equal significance for the process. In particular, the selection of a 
good chairperson for the panel was seen as conducive to the overall success of the team. 
Finally, it was emphasised by the participants that the vast majority of expert panels 
cooperate well and fulfil their assigned tasks, and that the evaluated institutions are 
generally content. 

The questions about availability and resources are also relevant in this discussion. 

It was agreed that ideally, assessment training and clear instructions are to be used 
in addition to the panel members’ expertise in the field. There was discussion at the 
workshop about at what stage of an academic career would such training be the most 
beneficial. 

Expert knowledge in public 

In Europe the national assessments of higher education normally result in public reports. 
The public reports are seen as important information to students, authorities and society 
in general. The Standards and Guidelines for Quality Assurance in the European Higher 
Education Area (ESG) state that “the interests of society in the quality and standards 
of higher education need to be safeguarded”. According to the ESG, “Reports should be 
published and should be written in a style, which is clear and readily accessible to its 
intended readership [. . .]”. It is generally assumed that the reports are important for the 
evaluated institutions’ reputation. But what “knowledge” or “ authority ” status do (or 
should) the reports legitimately have? (Workshop announcement) 



To introduce the above mentioned session, Karl Dittrich, the Chairman of the 
Accreditation organisation of the Netherlands and Flanders (NVAO), presented the 
variety in European practices on the publishing of review reports. The presentation 
was followed by a panel discussion by Norwegian higher education institution (HEI) 
representatives: Ernst Hakon Jahr, former Rector, the University of Agder; Torunn 
Klemp, Rector, Sor-Trondelag University College and Ingrid Torsteinson, the Head of 
Nursing Department, Bergen Deaconess University College. 

At the workshop, it was pointed out that the panel is and should remain responsible 
for the contents of the review report. It seemed that the extent to which the agency was 
involved in the technical drafting of the report varied from country to country. 

From the point of view of the institutions, as exemplified by experiences in Norway, 
the message was that regulations concerning the work of the panel need to be strictly 
clarified in advance. The panel should be as clear and specific as possible in their 
reports. In addition, review reports always have a political context that should also be 
considered. 

External reviews were acknowledged to have positively supported the strategic work 
of the institutions. However, serious problems about the legitimacy of the panel and its 
report were identified by the HEI representatives. 

The role of the media was discussed; it was noted that sometimes the media distorts 
reports, and presents the goals for improvement as failures. This can be a problem for 
the public reputation of a higher education institution. In addition, it may lead to the 
panel’s reluctance to express strong opinions. In public discussion, there is a tendency 
to emphasise the negative, but as it was reminded at the workshop, most higher 
education institutions do perform very well. 



Chapter 2: On the use of expert 
knowledge in national evaluations 
and quality audits 

Berit Askling, University of Gothenburg 

2.1 Controlling quality in higher education - before and now 

Over the years, the quality of higher education has been controlled in various ways. The 
general development has been from academia’s own control of individual qualifications 
and performance by means of external expert teams (the so-called peer review model) 
to today’s national arrangements by the state, by which national quality assurance 
agencies control the way in which the higher education institutions work and perform. 

As long as the primary task of higher education was to produce the next generation 
of academics, quality control remained the task of academia itself - without any 
interference from the outside 4 . Academia could be regarded as a national and 
international ‘community of scholars’ with a customary responsibility for its own 
quality assurance and this control was mainly directed at the performance of individual 
academics (through examinations, habilitations, appointments). 

Today, national quality assurance agencies conduct national quality controls on a 
comprehensive scale. Established and regulated by the government, agencies assess 
the ability of institutional managements and organisations to develop and ensure 
the quality of entire educational programmes, disciplines as well as of single courses. 
However, assessing quality is no easy task. There hardly exists such simple correlation 
between measurable indicators and the quality one wants to measure that would allow 
for assessments by objective criteria alone. The agencies therefore have to arrange their 
evaluations in ways that are sufficiently sophisticated for the evaluated parties to accept 
them. 

According to Brennan and Shah (2000), the evaluating agency, as a tool within the 
system of a national government, must have a ‘bureaucratic’ authority. The specific 
competence of the agency’s own staff is the administration and conduct of assessment 
procedures, but their legitimacy does not extend to the performance of the very 
assessments themselves. For this purpose, the agency needs to employ expert ‘peers’ 
from academia itself - albeit now acting under the auspices of the agency. In the 
Scandinavian countries these academic experts are commonly joined by students, and 
often even by representatives of external stakeholders. 

An important question concerns the relation between the agency’s project managers 
and the academic experts of the review team. Who is the expert in and in which field? 
What possible contributions can the project manager make and what remains as the 
task of the ‘hired’ academic experts in the assessment procedure? 



4 See Brennan & al, 1994. 



2.2 National quality control as a means of national governance - a backdrop 

Since the early 1990s there has been a development towards increased decentralisation 
and deregulation in the higher education systems of many European countries. 

Under the influence of the New Public Management-oriented theories 5 the division 
of responsibilities between the state and its universities and colleges has shifted in a 
number of ways. Detailed governmental regulations and guidelines have gradually been 
replaced by self-regulation and output control. The institutions’ room for manoeuvre 
has been considerably widened in questions that relate to the content and organisation 
of study programmes, the recruitment of students and teachers, management and 
internal organisation as well as finance. At the same time, demands for various forms 
of accountability and reporting have increased. The extension of self-regulation 
is balanced against new instruments of control as accreditation, quality assurance 
assessments (audits) and programme or discipline evaluations. 

In addition, higher education has gone through a period of vast expansion since 
the early 1990s, while state financing has failed to grow at a corresponding rate. 
Dependence on external financing has increased. International exchange has increased 
in importance. In this context, the quality of higher education, research training and 
research have been questioned. Much effort has gone into the attempts to define 
simple, measurable quality indicators, while at the same time the negative effects 
of relying too much on control by such indicators have also been highlighted. This 
development is very evident in the Scandinavian countries where much energy has 
been put on finding a balance between controlling and enhancement oriented measures 
in the national systems of external quality assurance. The Scandinavian agencies 6 try to 
operate on a neutral arena between the political “commissioning agent” and academic 
“performers”. 

These agencies have to follow regulations that are defined by the governments 
and they must observe international agreements among similar agencies (e.g. ENQA) 
in order to create conditions for transferability. With the construction of a national 
agency, a new category of civil servants has arisen with specialised skills and functions. 
They are to make sure that the evaluating experts work in accordance with an agreed 
understanding of given criteria (particularly in the case of evaluating quality assurance 
systems in relation to established criteria), that the evaluations follow a certain 
procedure and that reports are written in accordance with a certain formula. 

2.3 Different types of assessment - Example: the Norwegian national agency 
NOKUT 

ACCREDITATIONS 

According to the Universities and Colleges Act of 2005 7 , accreditation means an 
academic’ assessment on whether a higher education institution, or a study programme 
provided by such an institution, corresponds to a certain given standard, defined 
in terms of a number of criteria. The accreditation must be based on an ‘evaluation 
performed by external experts, appointed by NOKUT’. 



5 See e.g. Bleiklie and Henkel, 2006. 

6 In addition to NOKUT: the Swedish National Agency for Higher Education and the Danish Centre for Quality Assurance and 
Evaluation. 

7 Norwegian Act of 1 April 2005 no 15: Of Universities and Colleges, § 3—1. Accreditation of study programmes and institutions. 



For NOKUT to obtain legitimacy for its accreditations, the experts must perceive 
to be familiar with the actual object of assessment. Depending on their personal 
competence, their loyalty may extend in different directions: for expert recruited from 
academia this loyalty will probably be directed towards the specific norms and values 
of academia itself. For those experts who are recruited from the field of employment 
stakeholders the loyalty may be first and foremost towards the occupational field, 
future colleagues, clients, etc. 

PROGRAMME EVALUATIONS 

In this case, the object of assessment is a specific educational programme. The aim may 
be to establish whether the programme maintains an acceptable academic standard. Or 
it may be to assess certain aspects of the programme, for instance its compliance with 
nationally defined reform intentions or that it has followed up the changes in national 
curriculum guidelines (or similar ‘directives’). 

The legitimacy of the evaluation will depend on whether the various stakeholders 
accept the final report as reasonable and trustworthy. Trustworthiness is obtained 
through sound reasoning, based on defined and acceptable references, and an 
informative outcome. A broadly composed expert panel, reflecting different stakeholder 
perspectives, will be heterogeneous in terms of its specialist expertise. 

Within NOKUT itself such collective expertise can hardly be found, covering 
programmes in avast number of different discipline areas. This means that NOKUT 
cannot claim the same kind of ‘professional ownership’ of the entire assessment 
procedure as it may in the case of quality assurance evaluations. The distance between 
the specific competencies of the external experts and NOKUT’s general evaluation 
expertise is bigger in the case of programme evaluations than in that of quality 
assurance systems’ evaluations. 

DISCIPLINE EVALUATIONS 

Discipline evaluations are in many respects similar to programme evaluations, only 
with a more exclusively ‘academic’ object. The group of stakeholders will probably be 
limited to academia itself. The task of the agency will therefore be somewhat simpler 
than with programme evaluations. The distance between the respective competencies 
of the experts and the agency will probably be even bigger than with programme 
evaluations. 

EVALUATIONS OF QUALITY ASSURANCE SYSTEMS 

Concerning evaluations of (internal) quality assurance, the object of the assessment 
is the institution’s quality assurance system and its function as a steering and 
management tool, being a recent addition to the set of evaluation types used in higher 
education. This type of evaluation did not arise and develop within academia itself; its 
history is brief and extra-academic. 

The quality assurance system of a university or a college is a control instrument that 
was introduced by the state. Governments are the main stakeholders, and possibly 
also the top institutional management and leadership groups, who acknowledge the 
importance of playing by the new rules. The national agencies have contributed actively 
to the promotion of the quality assurance systems of various countries. Experts who 
are engaged in these kinds of assessments must be familiar with theoretical models 



that regard higher education as a societal institution (with a politically defined mission 
and stakeholder expectations, etc.), as an organisation (with a network of horizontal 
and vertical lines of decision-making and information), as a vehicle for producing and 
disseminating knowledge (with short-term and long-term outcomes and effects that are 
difficult to measure). 

The experts are, in a sense, cognoscenti in relation to quality assurance systems 
but they should also be experienced and respected individuals in questions relating 
to the steering and management of higher education. The agencies’ officials, on the 
other hand, may be said to have expert knowledge within the specific area of national 
quality assurance systems, developed through their work in these agencies. They may 
assume an active role in their function as secretaries to the expert panels. They have 
a collective familiarity with quality assurance systems as evaluation objects that the 
other experts can hardly have. In an assessment panel the panel’s secretary takes on the 
responsibility for the entire procedure on behalf of NOKUT as an agency; he/she should 
therefore have a more active role in the evaluations of QA systems than in the other 
assessment types mentioned above. The procedure, representing something entirely 
new within academia, has been developed by NOKUT; the criteria are defined and 
decided by NOKUT (after having been heard by the universities and colleges, though) 
and the reports are written after a fixed template. 

2.4 Conclusions 

Except for the differences that have been pointed out above, there are also great 
similarities between the assessment tasks, irrespective of the type of evaluation. In 
order to be able to contribute constructively in committee work, and also to derive 
personal satisfaction from participating in it, the expert should: 

• be intimately familiar with research and teaching at the level of higher education; 

• possess a certain analytical talent - with natural curiosity and ability to be 
amused by ‘jigsaw puzzles’; 

• be able to draw on personal experience (as a teacher, researcher, administrator 
and/or manager) from university and college institutions; 

• be a good listener, be able to ‘read between the lines’ and catch ‘hidden’ meanings 
underneath the surface of statements; 

• be motivated to learn more about the ways in which higher education and its 
institutions work and operate under given conditions (political, economic, 
societal, etc.). 

To participate in assessment panel work implies committing oneself with one’s loyalty 
to academia to carry on the tradition of ‘peer review’ under new conditions, now in the 
service of a national agency. At the same time, the expert makes a contribution towards 
creating legitimacy for the assessments of the national agency. Through their various 
types of evaluations the national agencies are promoting the growth of a new kind 
of academic ‘professionalism’ through academic ‘experts’ or ‘peers’ who will uphold a 
systemic perspective in addition to their specific disciplinary perspectives. 

But the national agencies are also themselves developing a new kind of 
professionalism - in the form of national evaluation managers. To work as an evaluation 
officer in a national agency implies that your loyalty lies with the task and mission of 
the agency. Evaluation officers must develop a familiarity with the agency’s procedures 
for the different evaluation types and the different sets of criteria that are applied. They 



must strive to function as ‘national moderating judges’ in the evaluations, so that the 
assessments will not be biased by the composition of competences and perspectives in 
each individual expert group. They are finding their natural place in an international 
community of professional quality evaluators. 
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Chapter 3: Evaluation performed by 
the external expert panel 

Peder Haug, Volda University College 

3.1 Introduction 

In most countries of the European Higher Education Area (EHEA) higher education 
institutions and programmes are being evaluated. The results of these evaluations can 
be crucial for institutions and individuals. An evaluation can conclude that institutions 
will not be allowed to offer study programmes; that they lose an accreditation they have 
had earlier; or that they might have to make changes in study content, organisation and 
staff. 

It constitutes a real challenge if an institution is accredited on false grounds, 
when in reality it cannot offer sufficient educational and substantial quality. The 
collected information must be trusted, and the analysis and conclusions drawn from 
this information have to be in accordance with the data. This article will discuss the 
trustworthiness of evaluations from the viewpoint of the external expert panels. 

The expert panels are used in several evaluation models. A selected panel of 
specialists is responsible for the evaluation. The experts are often assisted by staff 
from the national agencies. This approach combines self-evaluation with brief site 
visits to the institutions by representatives from the expert panel. The experts conduct 
discussions and interviews with selected students, lecturers, researchers, administrative 
staff and people from the course, programme or institution under evaluation. In 
addition, a certain amount of existing data about the institutions, studies and courses is 
collected by the panel. However, there have been cases where studies providing a lesser 
amount of information have been used to support an evaluation. 

3.2 Background 

The approach to the subject of this article has been formed from a threefold 
perspective: 

- Firstly, the Norwegian Agency for Quality Assurance in Education (NOKUT) 
performed an evaluation of Norwegian teacher education in 2006 (NOKUT, 
2006), the methodology and approach of which were highly interesting; 

- Secondly, the author has been working in higher education and teacher 
education as administrator, lecturer and researcher for more than thirty years 
and is therefore well acquainted with the field; 

- Thirdly, the author has participated in the evaluation of teacher education in 
Norway, Sweden and Denmark by applying the same expert panel method used 
by NOKUT in 2006 and thus has gained a broad experience from this type of 
exercise as well as from research-based evaluations. 

3.3 The Arguments 

It can be argued that the expert panel approach has potential. There are, however, 
cases when it is practiced in such ways that do not provide sufficiently valid or reliable 



information on which to base trustworthy conclusions about the quality of a given 
study programme. There is clearly room for improvement. 

The expert panel methodology is better suited to drawing conclusions about some of 
the issues in an evaluation of study programmes than about some others. A distinction 
can be made between structural quality, process quality and result quality. Structural 
quality describes the institutional conditions, organisation and resources of a given 
institution or programme. Process quality is defined by what is going on inside the 
higher education institutions, primarily in the sector of teaching and learning. Result 
quality is what is achieved as a consequence of the processes and structures. The expert 
panel approach is most reliable when it comes to questions concerning the structural 
elements of a programme or of an institution, for instance the number of staff, their 
competence and grades; the number of students, accomplishments, programme 
organisation, systems for control of quality, etc. To answer process and result questions, 
other models of evaluation could be more appropriate, for instance research-based 
evaluation. Reasons for these conclusions will follow. 

3.4 The external panel approach data 

The quality of an evaluation is first and foremost dependent upon good and clear data 
and information. The most problematic aspect of any evaluation is formed by clear 
strategic institutional interests associated with an evaluation. It is in the institution’s 
best interest to appear as structured, competent and as professional as possible. This 
could affect how the self-evaluation is constructed and how the study programme, staff 
and organisation are presented by the institution. 

As a rule, in the expert panel approach, the institution officially approves the content 
of the self-evaluation, usually written by the self-evaluation group made up of the 
higher education institution’s relevant staff members, and presents it as its own. This 
does not necessarily mean that the document gives a description of how the institution 
and a programme actually appear and function. The picture given can be enhanced to 
fit what the institution strategically wishes it to look like. 

In the external expert panel method it is difficult for the experts to see beyond the 
data. The panel has limited access to the information about how the self-evaluation 
process has been conducted and how it has been handled within the institution. The 
expert panel has poor control over the data and the self-evaluation group members 
which provide that data, which reduces the validity of the evaluation. This lack of 
control makes itself apparent in several different forms. 

To trust the data, one should ideally know and be able to document who the 
self-evaluation group members are and how they were selected. It must be also made 
possible to verify systematically that the group sample was chosen to represent the 
potential respondents. It must be guaranteed, as well, that the respondents have been 
free and independent to answer according to their conviction and that their answers 
cannot be held against them. In the evaluation reports, these matters are often not 
described or emphasised. 

It has been witnessed in some cases that the most critical members of staff have not 
been allowed to put forward their views, and that their opinions have been suppressed 
or even denied. There have also been cases where they have not been invited to meet 
the expert panel for discussion. It has happened in certain cases that those staff 
members meeting the experts have agreed beforehand about the things to say, the 



things not to be said and who should be responsible of saying what. On one occasion 
the students were even told how to dress, how to sit and how to speak properly to the 
experts. 

When evaluating, it is also of importance for validity to document on what grounds 
the questions asked have been chosen and formulated. It is also important to make 
sure that the questions are relevant and cover the whole matter under examination. 

The panels have few possibilities to influence the questions asked and the perspectives 
given. The self-evaluation manual is to some degree standardised so as to be of use for 
many purposes, in different types of organisations and subjects. It is usually sent to the 
institutions before the expert panel has started to function. Because of the standardised 
form of questions in the manual, the theoretical basis taken into consideration and the 
references to existing research results as a basis for questioning are at best diffuse and 
at worst nonexistent. 

The institutions also put a different amount of effort into answering the questions 
in the self-evaluation manual. Some self-evaluations reports are very thorough, 
with extensively detailed presentations of both good and not so good aspects of the 
institution and its study programmes. Others can appear to be relatively superficial 
and do not present or discuss any circumstances that can provide information about 
problems or difficulties. There are examples that a self-evaluation has become a part of 
an internal struggle for power and hegemony, where one party in a discussion has won 
through with his/her views. 

Institutional site visits lasting for a day or two are too short to compensate for all 
these weaknesses. During the visit, the panel of experts meets five to six different 
categories of staff, for approximately an hour each. Often the group that meets the 
panel is large, between 5 to 10 persons, each of them with special responsibilities and 
understandings of the matters being studied, making it difficult to follow up questions 
because of the limited time available and the aim that everyone should be heard. 

In general, the questions that are easiest to answer are those concerning structural 
issues. Investigating processes and results presupposes intense questioning that is 
much more difficult to do in the expert panel method, but which is most relevant in 
defining programme quality. Questions about process quality and result quality are 
more difficult to answer and they may also be more controversial. The expert model 
functions much better when the task is accreditation, according to formal structural 
standards, and not the evaluation of education processes and outcomes. 

3.5 Power of explanation 

Explanations, in addition to descriptions of results, are not mandatory in research or 
evaluations. Explanations, however, could give additional information of great value, 
both when it comes to further understanding the issues under study in the evaluation 
process, and when recommending changes. The external expert panels are often 
supposed to present recommendations in cases where the required standards have not 
been reached, and they very often do. The point here is that evaluators could make 
more successful prescriptions, if and when they can, explaining the reasons behind 
poor evaluation results; explanations, therefore, are desirable. Evaluations following 
the expert panel methodology are low in power of explanation. If the description of a 
programme or a course is inaccurate, then any explanation is of little value. 



The collected data does not often lend itself to explanations and recommendations. 
The information gathered (self-evaluation, interviews and statistical data) is not 
suitable for deep analysis and explanation. In the data the facts about structural issues 
usually dominate, often because of the standardised question-manuals and procedures. 
The theoretical basis and references to research literature are also sparse, as mentioned 
already, meaning that the evaluation has not been based on an established knowledge 
tradition, making analysis much more uncertain. The danger then is that facts are 
taken at face value. 

In the evaluations of teacher education in Sweden and Norway, the consequences of 
this can be seen when structural facts are interpreted as data about processes or results. 

The most important source of information in the panel approach is the different 
experts’ own previous opinions and general knowledge about the field under review. 

The panel of experts must agree on how to interpret the data. Many persons are 
involved, chosen to represent a variety of backgrounds and interests. Each of them 
more or less has his/her own individual views on what does, and what does not count as 
interesting and important perspectives on the reviewed matter. 

What the panel is able to agree about then becomes the evaluation result, also 
referred to as inter-subjective understanding. The panel attempts to obtain valid 
conclusions within the frame of the group. The evaluation result then brings about 
discussions. Consensus can provide strong analysis and conclusions, but not necessarily. 
The “exercise” could lead to discordance and negotiations and be dependent upon 
tactics among experts more than on the arguments presented. Serious and profound 
conflicts within a panel are not uncommon. 

In the evaluation of teacher education in Sweden for instance, the chair of the panel 
wanted to resign because of a conflict of interest within the panel and because the 
national agency gave its support to one side in the discussion. In the evaluation of the 
Norwegian teacher education, the chair of the panel was dismissed. 

3.6 An inadequate method? 

What has been presented here thus indicates that there are requirements in the expert 
panel methodology that seem difficult to meet in practice. Without doubt, there is room 
for improvement in the way the methodology is put into practice. There are also several 
weaknesses in the methodology that could not be met by a better practice, but by 
applying alternative approaches. Without them, the risk is that the result is determined 
by the methodology, and not by the conditions in the evaluated institutions. 

The expert panel approach strongly emphasises internal data. That is data gathered 
from the institutions through self-evaluations and interviews with management, 
lecturers, researchers, students etc. What we learn is how staff and students perceive 
the education from the inside. We get a grasp of the internal debates and opinions 
about teacher education, and we are, to a much lesser degree, informed about how 
this education, under evaluation, functions in relation to practical teaching at an 
institution, as seen from the outside. 

Because of the main data, the expert panel has to operate within the same frames 
of reference as the institutions themselves (Hopmann, 2006). This is of course highly 
relevant when it comes to the questions of how the different institutions have organised 
the studies in accordance with the current regulations, which is a part of the structural 
quality. How this is done, however, does not in itself reveal anything about the process 



and result in quality of the education given. It seems that there is a risk of panels taking 
the information within the institutions too much at face value, without questioning it 
or relating this data to what is known about teacher education from other sources. This 
is also said to illustrate the fact that the recommendations made do not necessarily 
reflect what we know about the issues. I will illustrate this with some examples. 

First example: The evaluation of the Norwegian teacher education claims that a gap 
exists in teacher education between theory and practice, meaning that results from 
research are not adequately translated into practice, and vice versa. The expert panel 
concludes that this gap must be reduced. The philosophy of science tells us that this is 
questionable (Weinert, 1995). 

Second example: Research also shows that all students are critical to their own 
education. Students in teacher education are perhaps more critical than most, but they 
become less and less critical when they start working as teachers in school (Jordell, 
1989). It also counts that the most important review of teacher education research 
concludes that there is no evidence to tell us how to organise or develop teacher 
education so as to be able to ensure that it meets a certain standard. 

Third example: to explain the lack of quality, the panel emphasises the internal 
institutional dispositions more than the external conditions. Both in Sweden and 
in Norway the evaluations of teacher education did not stress the significance of 
recent contextual, practical and political conditions for teacher education. In both 
countries, recent reforms in teacher education had taken place without the necessary 
preparations. When trying to understand why teacher education did not live up to 
expectations then, the panels explained deviation as a consequence of institutional 
strategies rather than as the result of national policy. In research terminology the 
panels “went native”. 

3.7 Concluding comments 

The conclusion from the first section is that the evaluative description of an expert 
panel may be invalid, and that an expert panel approach is vulnerable to systematic 
data errors. The external expert panel has little control over data. 

The conclusion from the second part is that there is little information that makes 
it possible to explain and analyse the deeper reasons behind a description of a study 
programme. The method’s strength is in factual and structural information, and not in 
information about processes and results. 

The third part of the presentation concludes that the external expert panel method 
in all probability provides internal perspectives on what is studied; however, the 
method has difficulties in giving an external description of the institutions and study 
programmes. 
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Chapter 4: Experts’ recommendations 
and decisions, criteria for selection, 
publicity and mandate 

Anne Karine Sorskar 

Prior to the workshop: Assessing educational quality: Knowledge production and the 
role of experts, the participants were asked to answer a few questions on the tasks and 
status of experts in their own agencies. The purpose of this exercise was to get a picture 
of similarities and differences between the agencies in their practice with regards to 
expert knowledge, with reference to the situation in Norway. The questions were: 

1. What are the relations between expert evaluations and decisions in external 
quality assurance in your country? Are the relations affected by the types of 
procedure, e.g. accreditations, audit or other evaluations? 

2. Please send us the agencies written set of criteria for selection of experts. We are 
interested in the kind of competencies required for the various evaluations. 

3. Publicity and publication of evaluation reports: Are the evaluation reports of the 
agency public? If so, are the reports actively published, e.g. on the website of the 
agency? 

4. How detailed is the mandate for the various sort of evaluations? If possible, send 
us examples. 

The answers received can be interpreted in different ways, and during the workshop 
many of the agencies presented exceptions to their general rule. It was therefore a 
challenge to put all the agencies into clear-cut categories for the presentation of the 
results at the workshop. This article attempts to give a more nuanced overview even 
if there are some limits to defining the agencies and countries categorically due to the 
variations and plurality that exists between them. This means that not all the details 
found in the answers were put into categories which fully comply with the realities in 
the agency or country. 

Staff or experts from the following 21 agencies were present at the conference and 
completed the questionnaire: 

AAC, Austria; ACE, Denmark; ACQUIN, Germany; ANECA, Spain; AQA, Austria; 
AQAS, Germany; AQU, Catalunya; ASIIN, Germany; CTI, France; EHEAC, Estonia; 
EUA-IEP, Belgium 8 ; EVA, Denmark; evalag, Germany; FHR, Austria; FINHEEC, 
Finland; HQAA, Greece; NEAA, Bulgaria; NOKUT, Norway; OAQ, Switzerland; 
SKVC, Lithuania, VLIR, Belgium. 

Even if this sample cannot be regarded as representative of ENQA members as such, 
this article compares the results with those of a survey produced by ENQA and the 
Danish Evaluation Institute in 2003, Quality procedures in European Higher Education 9 , 
to which thirty-four agencies responded. 



8 IEP is the institutional evaluation programme with the European University Association (EUA). 

9 ENQA Occasional Papers 5 (ISBN 951-98680-8-9) 



4.1 Relations between expert evaluations and decisions 

Question 1: What are the relations between expert evaluations and decisions in 
external quality assurance in your country? Are the relations affected by the types 
of procedure, e.g. accreditations, audit or other evaluations? 

The answers from the agencies give reason to conclude that when an evaluation report 
from an expert panel is the basis for a decision, there is always some other body than 
the panel that makes the decision. There are also slightly different decision-making 
procedures depending on whether the evaluation is for audit or accreditation purposes. 
For both - accreditation of programmes and institutions - there are five categories of 
decision-makers, but only two categories for decisions on audit. 

Six agencies reported that they use quality audit. From 2008, a new type of 
evaluation, System Accreditation, has been introduced in Germany. All the details were 
not yet available, so the answers of the four German agencies are not included. 

Five of the six agencies reported that the agency itself is the decision-maker on 
quality audit, and one agency reported that the decision-maker is a governmental body. 
There are also different types of decisions that follow from a quality audit. 

Seventeen of the agencies undertake evaluations that are the basis for the 
accreditation of programmes or institutions. Seven of the agencies are themselves 
decision-makers for all their accreditations. For six of the agencies some of the 
accreditations given by the agency have to be confirmed by the ministry, e.g. 
accreditations of new programmes in public institutions and programmes leading to a 
national degree that should be awarded by the ministry. 

The most common decision-making body in the agency is a standing committee 
which holds the same competencies as the experts (e.g. academic staff, work-life 
representatives and students). An example of another decision-making body within the 
agency is CTI in France. CTI consists of thirty-two members, and the plenary of the 
agency makes all the decisions. The expert-panels are appointed among those thirty- 
two members, which mean that they also are included in the decision-making body. 
Another example is NOKUT, where the board of NOKUT is the decision-maker, but it 
has delegated to the director general the right of decision on some accreditations, e.g. 
initial accreditation of programmes at bachelor’s and master’s level. 

Two agencies reported that another agency was responsible for decisions on 
accreditation. For the OAQ in Switzerland the University Conference made decisions 
on universities, and a governmental body made decisions on universities of applied 
science. Table 1 shows the decision-makers of the agencies. 

TABLE 1: DECISIONS-MAKERS ON AUDIT AND ACCREDITATION 



DECISION MAKER 


AUDIT 


ACCREDITATION 


The agency 


ANECA, NOKUT, FINHEEC, AQA, 
AQU Catalunya 


FHR, NEAA, ACE, ASIIN, AAC, 
AQAS, ACQUIN 


The agency, recognition by the 
ministry on some of the decisions 




NOKUT, CTI, ANECA, SKVC, 
EHEAC, AQU Catalunya 


Another agency 




VLIR, AQA 


University Conference 




OAQ 


Governmental body 


OAQ 


OAQ 



There were also four agencies (EVA, EUA-IEP, HQAA and evalag) that did not use 
their evaluations as the basis for decisions. The institutions that were evaluated were 
themselves responsible for the follow-up from the evaluation. It should be noted that 
the ENQA survey of 2003 showed that about a half of the agencies, in addition to 
evaluation with an emphasis on enhancement, also made evaluations as the basis for 
accreditation. The report concluded that the agencies still and foremost performed 
their quality assurance and/or enhancement by the traditional way of peer-review. But 
that report also pointed out that there was an increased focus on accreditation. These 
new answers discussed here indicate that there has been a shift in the main focus since 
2003. 

Seven of the agencies (NOKUT, FINHEEC, NEAA, ASIIN, VLIR, OAQ and AQA) 
reported that in a few cases there were divergences between the recommendation 
from the expert panel and the decision. Only VLIR reported that they had a procedure 
for handling such divergences. NVAO is the agency that makes decisions on the 
basis of the evaluations undertaken by VLIR. If there is a discrepancy between the 
recommendation from VLIR’s expert panel and the decision of NVAO, then NVAO will 
organise a hearing with the expert panel and the institution. 

4.2 Criteria for selection of experts 

Question 2: Please send us the agencies’ written set of criteria for selection of 
experts. We are interested in the kind of competencies required for the various 
evaluations. 

Most of the agencies had requirements on the composition of the expert panel. A 
common composition was three higher education staff members, one work-life 
representative and one student. Many agencies also had requirements for different 
competencies that had to be represented in the panel, for instance international 
expertise of evaluations, experience in management of a higher education institution 
or an academic leadership position. These requirements could be considered for other 
agencies as the criteria for the selection of their experts. In other words, the same 
competence can be at the same time a merit or requirement in the panel and a criterion 
of selection of the individual member/expert. 

In table 2, 3 and 4 the column present in panel refers to what kind of competencies 
are required in any panel and the column criteria for selection refers to the criteria for 
selecting individual experts. These criteria mainly refer to the academic experts - not 
the student or the representative from work-life, although it would be desirable that the 
work-life representatives would also have similar competencies as the experts from the 
academia. 

The agencies reported on various compositions of the expert panel in different types 
of evaluations. There are mainly three types of evaluations: evaluation of systems 
of quality assurance (quality audit), evaluation of institutions and evaluation of 
programmes. In this article there is no distinction made between evaluations that are 
the basis for decision (accreditation) and those that are not. There is no indication that 
the profile of the experts or the composition of the panels differs in those agencies that 
are not undertaking accreditations. Some of the agencies emphasised that even when 
the evaluation is used as basis for accreditation, the report should also give a suggestion 



for improvement of the quality of the programme, even if such programme meets the 
standards required for accreditation. 

The expert panels are normally appointed by the agency, but in some cases the 
institutions can appoint some of the members of the panel. Several agencies mentioned 
that the institutions are given the opportunity to suggest members, or comment upon a 
proposal from the agency before the panel is appointed. 

4.2.1 QUALITY AUDIT 

All agencies (6) required higher education staff in the panel of a quality audit, and 
that at least one of them should be a full professor. All agencies required experience of 
quality assurance/evaluation methods (for FINHEEC this is a criterion) and that there 
is a student member in the panel. Five agencies required experience in management 
of higher education institution or academic leadership position, and one agency 
(FINHEEC) had this competence also as a criterion for selecting the individual expert. 
Three agencies required an international expert. Competencies mentioned by one 
agency were: member of the first audit panel for the institution in question, knowledge 
of the system of higher education in the country, and work-life representation. Two 
agencies required a balanced gender distribution in the panel. 

Five agencies have requirements of unbiased judgement. Only one agency 
(FINHEEC) has, in addition to requirements to the composition of the panel, written 
criteria for selecting the individual expert. These requirements were: knowledge 
of the system of higher education, experience of audit/evaluation, experience of 
management or quality assurance-systems and participation in the agency’s training. 
The requirements for the composition of panels for quality audit and selection criteria 
are shown in table 2 . 

TABLE 2: QUALITY AUDIT: COMPETENCIES REQUIRED 



COMPETENCIES 


PRESENT IN PANEL 


SELECTION CRITERIA 


HEI-staff incl. full professors 


All agencies (5) 




Students 


EVA, FINHEEC, NOKUT, OAQ, 
AQA 




Management of higher education/ 
academic leadership position 


AQA, OAQ, NOKUT, FINHEEC 


FINHEEC 


Work-life representative 


FINHEEC 




International expert 


NOKUT, EVA, OAQ 




Member of the first audit-panel for 
the institution in question 


OAQ 




Knowledge of the higher education 
system 


OAQ 


FINHEEC 


Experience of quality assurance/ 
evaluation 


EVA, NOKUT, OAQ, AQA 


FINHEEC 


Balanced gender distribution 


NOKUT, AQA 




Participation in the agency’s 
training 




FINHEEC 


Unbiased judgement 




FINHEEC, NOKUT, AQA, OAQ 



4.2.2 EVALUATION OF INSTITUTIONS 

All agencies (11) required higher education staff in the panel, and that at least one of 
them should be a full professor. Four competencies were mentioned by more than five 
agencies, either as a requirement for the panel or as a selection criterion: 

• management of higher education institution or academic leadership position (8 
agencies) 

• knowledge of the system of higher education in the country (8 agencies) 

• quality assurance expert or experience of evaluation methods (8 agencies) 

• international expert (7 agencies) 

Another seven competencies were required or used as selection criteria by less than five 
agencies: 

• work-experience (4 agencies) 

• student (2 agencies) 

• experience in the development and design of programmes (2 agencies) 

• expert in teaching/learning methods (2 agencies) 

• academic reputation (2) 

• international experience (1 agency) 

• knowledge of educational trends (1) 

• personal qualities (1) 

Two agencies required balanced gender distribution in the panel, and nine agencies 
have required unbiased judgement. The requirements of the composition of panels for 
evaluation of institutions are shown in table 3. 

TABLE 3: EVALUATIONS OF INSTITUTIONS: COMPETENCIES REQUIRED 



COMPETENCIES 


PRESENT IN PANEL 


SELECTION CRITERIA 


HEI-staff incl. full professors 


All agencies (11) 




Management of HEI/academic 
leadership position 


OAQ, FHR, AQA, evalag, NOKUT, 
HQAA 


EUA-IEP, HQAA, AQA, AAC, 
evalag 


Work-life representative 


evalag, HQAA, NOKUT, FHR 




International expert 


NOKUT, evalag, HQAA, FHR, 
EHEAC, AQA, OAQ 




Experience in development & 
design of programmes 


A AC, evalag 


AAC, evalag 


Knowledge of the system of higher 
education 


OAQ, FHR, AAC, AQA, evalag, 
SKVC 


NEAA, EUA-IEP, evalag 


Experience of quality assurance/ 
evaluation method 


evalag, NOKUT, AAC, OAQ 


HQAA, ASIIN, NEAA, EUA-IEP, 
evalag 


Balanced gender distribution 


NOKUT, EUA-IEP 




Student 


EUA-IEP, NOKUT 




Expert in teaching/learning 
method 


evalag, AQA 


AQA, evalag 


International experience 


evalag 


evalag 


Academic reputation 




NEAA, evalag 


Unbiased judgement 




OAQ, NOKUT, AQA, AQU 
Catalunya, evalag, HQAA, NEAA, 
SKVC, FHR, 


Knowledge of educational trends 




EUA-IEP 


Personal qualities 




NEAA 



4.2.3 EVALUATION OF PROGRAMMES 10 

All agencies (17) required higher education institution staff in the panel in evaluation 
of programmes, and at least one of them should be a full professor in the discipline in 
question. Some agencies required all panel members to be professors. 

Four types of competencies were mentioned by more than half of the agencies: 

• experience of evaluation method/quality assurance (13 agencies) 

• work-life representative (9 agencies) 

• management of higher education or academic leadership position (8 agencies). 

• student (7 agencies) 

Competencies mentioned by less than half of the agencies: 

• experience in development and design of programmes (6 agencies) 

• knowledge of the system of higher education (6 agencies) 

• international expert (5 agencies) 

• expert in teaching/learning methods (4 agencies) 

• international experience/knowledge of the Bologna process (4 agencies). 

• academic reputation (2 agencies) 

• personal qualities, e.g. communicative skills and goodwill (2 agencies) 

• experience of international developments in the discipline (1 agency) 

• educational expert (1 agency). 

Two agencies required a balanced gender distribution in the panel, and eleven agencies 
have requirements for unbiased judgement. Two agencies required participation of the 
agency’s training. Several agencies mentioned that they had (annual) training of the 
experts, but only a few reported that this training was mandatory for the experts. 



10 There are somewhat generalised conclusions as some agencies had a more nuanced criteria for initial accreditation and evaluation 
of a given accreditation. 



TABLE 4: EVALUATION OF PROGRAMMES: COMPETENCIES REQUIRED 



COMPETENCE 


PRESENT IN PANEL 


SELECTION CRITERIA 


HEI- staff incl. full professors 


All agencies (17) 11 




Management of HEI/academic 
leadership position 


FHR, AAC, CTI 


HQAA, AAC, AQU Catalunya, 
FINHEEC, ASIIN, evalag 


Work-life representative 


ANECA, ACQUIN, FHR, AQA, 
AQU Catalunya, CTI, EVA, 
FINHEEC, VLIR 12 




International expert 


NOKUT, FHR, EHEAC, OAQ, 
VLIR 13 


VLIR 


Experience in development & 
design of programmes 


AAC, FHR 


AQAS, AAC, AQU Catalunya, 
ANECA, evalag 


Knowledge of the system of higher 
education 


FHR, AAC, AQA, SKVC, CTI 


FINHEEC 


QA-expert or experience of 
evaluation methods 


NOKUT, AAC, FHR, AQA, EVA, 
VLIR, SKVC, AQU Catalunya 


evalag, ASIIN, AQAS, FHR, 
ANECA, AQU Catalunya, 
FINHEEC 


Balanced gender distribution 


NOKUT, VLIR 




Student 


ANECA, VLIR, ACQUIN, ASIIN, 
AQU Catalunya, FINHEEC, 
NOKUT 14 




Expert in teaching/learning 
method 


AQA 


AQAS, ASIIN, evalag 


International development of the 
discipline and educational expert 


VLIR 




Unbiased judgement 




OAQ, NOKUT, AQA, FHR, CTI, 
AQAS, AQU Catalunya, VLIR, 
SKVC, ACQUIN, ASIIN 


Academic reputation 




AQAS, evalag 


Participation in the agency’s 
training 




FINHEEC, VLIR 


International experience/ 
knowledge of the Bologna process 




AQAS, evalag, FINHEEC, VLIR 


Personal qualities 




AQAS, SKVC 



4.2.4 EXPERTS’ COMPETENCIES 

The overview does not show significant differences between the agencies regarding 
the competencies for the panel or for the individual expert. Most agencies have 
different profiles for the expert panels depending on the type of evaluation. Most 
agencies undertake evaluation of programmes and about half of them make evaluation 
of institutions and/or quality audit. There were only three agencies (NEAA, HQAA 
and EUA-IEP) that only made evaluations of institutions (and not programmes). Five 
agencies were making quality audits, and the four German agencies will most likely 
introduce this type of evaluation during 2008. 

The ENQA survey of 2003 concluded that most often the experts in the panels 
represent the area (subject). An international expert was very often represented in 
these panels, and students were included in a few cases. In addition, there were also 
personnel from the quality assurance agencies in around 40% of the panels. 



11 ACE Denmark is not mentioned here, since they had not yet decided by February 2008 what competencies their experts should 
have. 

12 If relevant to the programme. 

13 Under the condition that he/she speaks Dutch if the assessment proceeds in Dutch. 

14 NOKUT uses students in general in evaluations and revisions of given accreditation, but not in initial accreditations. 



The workshop of 2008 showed that the requirements for the competencies of a 
panel or for the selection criteria of experts were quite similar for evaluations of 
programmes and evaluation of institutions. In both types of evaluations all agencies 
required higher education staff members (professors) and members with experience 
of quality assurance/evaluation. It seems that the agencies in 2008 require the higher 
education staff also to have experience from quality assurance or evaluation methods. 
Staff members from the quality assurance agencies seem to take another role than the 
experts. They are facilitating the evaluations, but are not regarded as panel members. 

In the evaluation of institutions (and also in audits) most agencies use management 
of a higher education institution/academic leadership position either as a selection 
criterion or as a requirement to be present in the panel. It is also an important selection 
criterion for experts in the evaluation of programmes. This criterion was not mentioned 
in the 2003 ENQA report. Knowledge of the system of higher education is also a 
competence that is often present in panels evaluating institutions, and not mentioned 
in the 2003 ENQA report. 

About half of the agencies use work-life representatives and/or students in their 
panels, and several other agencies are considering doing so. Students are most likely 
to be found in an audit-panel but in these panels we hardly find any work-life 
representative. Only a few agencies use students in panels that are evaluating 
institutions, but here a work-life representative is more likely to be found. In the 
evaluation of programmes, many agencies (8) use a work-life representative, and 
students are used by seven agencies. 

Some agencies still compose their panels for evaluating programmes exclusively with 
peers from other higher education institutions. For these agencies, both requirements 
for the composition of the group, as well as the selection criteria were likely to include 
(1) experts in teaching/learning method, (2) experience of international development in 
the discipline, (3) educational expert and (4) experience in development and design of 
programmes. It seems that even if the panel members are experts in the academic field 
in question, the agencies have elaborated on making additional requirements for these 
experts. 

Many agencies intend to use international (foreign) experts, but it may be concluded 
that there has not been an increase in the use of international experts since 2003. The 
answers from the agencies indicate that language is a barrier since in most cases the 
national language is the language of the evaluation. On the other hand, for EHEAC 
in Estonia, the only written requirement was that at least two nationalities should be 
present in the panel. SKVC in Lithuania undertakes its evaluations in English, which 
makes it easier to recruit experts from abroad. In Germany, some agencies require 
that the experts should come from another state than the assessed higher education 
institution. International experts seem to be more often present in the evaluations of 
institutions than in the evaluations of programmes. 

It could be said that the criterion “unbiased judgement” represents the legitimacy 
of the evaluation. This criterion is also the most frequent requirement for selection of 
experts. This requirement could be expressed in different ways, e.g. “written expression 
of non-conflict of interests with the evaluated institution”, “the members must not 
allow themselves to be led or influenced in their judgement by persons or parties at 
the institution or study programme or by other parties involved”, or “must be able to 
distance themselves sufficiently from personal ideas, convictions or preferences with 



respect to the academic area/the domain/the discipline involved in the assessment”. 
AQAS has expressed an exclusions criteria defining that the panel members should not 
have been involved with the institution in question. 

This overview could lead to the conclusion that the typical expert recruited by the 
agencies still is a professor, but it is likely that he/she has experience of management 
of higher education institutions or holds an academic leadership position. Through 
experience gained with evaluation he/she would also have achieved competencies 
in evaluation methods. In addition, the expert would attend training offered by the 
agency. That leads to the conclusion that the typical holder of expert knowledge is a 
professor with experience in management of a higher education institution and quality 
assurance/evaluation. He/she holds slightly different competencies than the traditional 
peer. Expert knowledge therefore tends to comprise more than peer knowledge. 

4.3 Publication practices 

Question 3: Publicity and publication of evaluation reports: are the evaluation 
reports of the agency public? If so, are the reports actively published, e.g. on the 
website of the agency? 

Table 5 attempts to arrange the answers from the agencies into four categories. 

Please see the article from Karl Dittrich (NVAO) for a more nuanced and in depth 
interpretation of the differences between the agencies and what consequences they 
might have on the evaluation process. 



TABLE 5: DIFFERENT TYPES OF PUBLICATION PRACTICE 



To publish all decisions and reports 


ACE, ANECA, HQAA, FINHEEC, VLIR, AQA, AAC, 
EVA, AQU Catalunya, EHEAC, SKVC, NOKUT 


To publish the decisions or summary of the reports 


ACQUIN, AQAS, FHR, NEAA, CTI 


To publish only positive decisions 


OAQ 


Public to a small extent 


EUA-IEP, evalag 15 , ASIIN 



Most agencies reported that they publish their reports or summaries of them, as well 
as the decisions taken. However there are some exceptions to the general rule. NOKUT 
exemplifies how difficult it is to categorise some of the agencies. It actively publishes all 
reports on audit (regardless of the outcome), but the reports from an expert committee 
that have evaluated (ex-ante) a programme as basis for accreditation are to be found on 
the website (regardless of the outcome), but not actively published. In Norway, a public 
body must have certain reasons not to make a certain document public. It might thus 
be illegal not to make an evaluation report public. But that does not mean that they 
should be actively published. This distinction between making a report public and in 
publishing it is not mentioned by other agencies. 

There are three agencies that do not publish their reports and decisions. Two of 
them are among the agencies that do not use their evaluations as basis for decision. 

Also AQA, which normally publishes all its decisions and reports, informed that it 
does not publish reports that are not a basis for accreditation. ACQUIN is an example 



15 evalag and EAU-IEP informed that the institutions decide themselves if the reports should be published. 



of an agency publishing the decisions, but leaving it to the institution to publish the 
report. ACQUIN is among the agencies that emphasises that the reports should be the 
documents providing advice for improvement. That can lead to the conclusion that 
evaluation reports that are not the basis for an accreditation or recognition decision, are 
less likely to be published than reports that have such a function. 

Only one agency, OAQ, publishes exclusively the positive decisions. However, this 
agency informed that there is a draft of a new law at the moment that foresees the 
publication of all reports. Other agencies also reported on changes in policy towards 
more publicity. CTI reports that they only publish the decisions, but that there is 
an ongoing discussion on whether the reports also should be published. EUA-IEP 
informed they are encouraging the institutions to publish their reports, and that the 
institutions are also increasingly doing so. 

4.4 Mandate for evaluations 

Question 4: How detailed is the mandate for the various sorts of evaluations? If 
possible, send us examples. 

Although the answers were too scant for comparing the content of the mandates, a few 
observations can be made about the answers to this question. 

Fourteen agencies reported having a mandate, procedures and guidelines or manuals 
to guide their experts in their work. Six out of these fourteen agencies sent examples. 
Three agencies said they were preparing mandates while four agencies gave no answer. 

Based on the foundation of the answers, it could be concluded that the agencies 
have, within a legal framework, developed detailed mandates for their various types of 
evaluations. Most of the agencies have their mandate stated in national law. In some 
countries the government gives more detailed regulations, e.g. in Spain the functions 
of evaluations and the role of ANECA are stipulated in the national university act. A 
royal order that stipulates official degrees gives further directives for the mandate of 
ANECA. In Estonia, the ministry regulates EHEAC. Austria is an example of a country 
where an agency, FHR, gives the general regulation on evaluation. One agency, OAQ in 
Switzerland, has its mandate from the Swiss University Conference. 



Chapter 5: External Experts in 
Quality Audits: Knowledge, decisions 
and credibility 

Kirsten Hofgaard Lycke 

5.1 Abstract 

The role of experts is vital to the quality of national external audits. If the audits are 
to be considered as consistent and credible the external experts need to perceive the 
context in the same way, have similar orientations towards their task and implement 
the same practice. The present data on the perceptions of quality assurance experts 
demonstrates that there is a broad agreement on the expert knowledge and procedures. 
Divergent views, however, indicate aspects of audit processes that might need revision 
and prompt discussion as to how the auditors’ role might be enhanced. 

5.2 Introduction 

The effectiveness of the current formative approach to quality assurance is highly 
dependent on the professionalism and independence of the external experts who do the 
evaluations 16 . Yet little is known about how these experts perceive and carry out their 
role or whether variation in this respect influences the interpretations and evaluations 
that are made 17 . What competence and knowledge do experts bring to their task? How 
are decisions made? How do experts perceive the credibility and usefulness of the audit 
reports? 

5.3 Background 

National systems of quality assurance have become a part of the higher education 
sector in a number of countries. In a relatively short time audits have achieved an 
important standing. The audit recommendations are often regarded as directives. The 
audits are commonly carried out by external experts appointed by the national quality 
agencies. The credibility of audit processes and results rely largely on the competences 
of the external experts. For audits to be generally accepted and not be regarded as 
reliant on the whims of different auditors, it is necessary to have a high degree of 
communality in the way that auditors perceive their mandate, carry out audit processes 
and even in the perspectives that they bring to their task. A review of the literature 18 
indicates that there is little known about the selection, work and perceptions of the 
experts who act as auditors. The study presented here is an attempt to gain such insight 
about the external experts who act as auditors for the Norwegian Agency for Quality 
Assurance in Education (NOKUT). 



16 Purser 2007 pl36 

17 Elken 2007, Aas et al 2007 

18 Elken 2007 



5.4 The Norwegian audit process 

The Norwegian audit process follows the general model known and applied 
internationally in the last decade 19 . According to NOKUT 20 the purpose of institutional 
quality assurance systems in Norwegian higher education is: 

• to ensure that educational activities are of high quality and are developing toward 
further improvements; 

• to reveal cases of deficient quality and to detect good and bad quality; 

• to provide the institution with a basis for self assessment and change; 

• to help develop a strong quality culture. 

The mandate for external experts in the Norwegian national audit process is related to 
this purpose and is stated as follows 

• to work in accordance with the Ministry regulations and NOKUT’s criteria; 

• to deliver a written report on their findings and conclusions; 

• to offer a clear recommendation to NOKUT as to whether the audited system 
should be approved; 

• to give the higher education institution advice on how to enhance their quality 
work. 

From the external experts’ point of view, the audit process has four distinctive phases: 

Preparation: The experts receive and review documentation of the institutional quality 
system individually in lieu of the NOKUT criteria. There are no set procedures for how 
the experts should collaborate in the preparation phase. It is common for the expert 
panel to have a 1-2 hour meeting at the site to prepare as a group. Contact with the 
institution including practical aspects of the later site visit, the site visit program and 
information about documentation, as well as coordination of the expert panel, are 
undertaken by an officer from the agency. 

Site visit: The external experts participate in site visits as members of an expert 
panel. The visit usually lasts for 2-4 days depending on the size and complexity of 
the institution. The panel conducts group interviews (45-60 minutes) with teachers, 
students, and support staff and management at all institutional levels. There is usually 
a preparation session of one to two hours before the interviews and short breaks for 
discussion between the interviews. An additional hour is usually added at the end of 
each day to sum up impressions and agree on how to approach further. 

Decisions: Based on the institution’s documentation and the information gathered at 
the site visit, and by using their expertise as a frame of reference, the external experts 
are to offer a clear recommendation to NOKUT as to whether an audited system should 
be approved. The experts are also to give advice on how the institution can enhance 
its quality work. The final decisions are made after the site visit preceding a thorough 
review of all data and materials. However, decisions and recommendations are usually 
the underlying themes in the panel discussions at the site visit. 



19 Stensaker 2003, ENQA 2005. 

20 2005 



Report: The expert panel is obliged to deliver a written report on its findings and 
conclusions. The current practice 21 is to organise the report according to the ten agency 
criteria, to demonstrate systematically how the criteria are met by the institution and 
how the conclusions of the panel are reached. The process of writing the report is the 
responsibility of the expert panel, usually coordinated by an agency officer. The agency 
board makes its decision on the basis of the report, but it is free to depart from the 
recommendation of the expert panel. The experts cannot participate in public debates 
about their assessments or the content of the report until the board has made its 
decision known. 

The structure of the process indicates that the experts are to apply their professional 
judgement on the quality of the institutions’ materials and activities. The experts’ 
professional judgment is reached through individual and panel considerations. 

A study of external experts: materials and methods 

The Norwegian agency has established a pool of approximately 45 experts to draw on 
for each separate audit. The selected experts have experience as academic leaders, 
developers and evaluators, and/or have an acknowledged scholarly reputation. The pool 
also includes students nominated by national student unions. 

A survey study of this pool of experts was conducted in April 2007. At that time a 
web-based electronic questionnaire was distributed to all 44 experts in the pool. 

Replies were submitted by 29 experts (response rate 67.4%). The questionnaire had 
questions (a non-cognitive questionnaire with graded and open questions) related to 
expert knowledge and experience, context and perceptions. 22 

5.5 Results 

The experts’ competence 

Of the respondents to the questionnaire, 70% were academics and 30% were students 
mirroring the composition of the pool of external experts. 88% had participated in 
audits three or more times. According to the respondents, 41% had experience as panel 
leaders. The formal qualifications of the external experts match the criteria for their 
role (see above). All respondents had a positive attitude towards the expert role in audits 
and expressed interest in the quality work of higher education institutions as well as the 
opportunity to gain general insight into quality systems and their development. 

When asked, the experts considered that their most important competences for the 
audit process were not so much their formal qualifications, but those competences that 
had been acquired through experience. The three most important areas of experience 
were: 

• developing quality in higher education; 

• reviewing and evaluating higher education institutions; 

• interpreting and applying different data sources. 



21 Lycke 2004 

22 The study was based on the SAKUproject, a project aimed to discuss and enhance the task of the external experts in quality audits 
for NOKUT (Aas et al 2007, Askling 2007) and discussions at NOKUT conferences for auditors (January 2006 and 2007). 



The decision process 

The work as external experts was experienced as positive by all respondents. The 
experts were satisfied with the conditions (time, information, documentation and 
progression of the work) for carrying out the audits. They were likewise satisfied 
with the role of the agency officers who participate as secretaries to the audit groups. 
According to the responses, most important to the process was the composition in the 
expert panels. The experts found the choice of panel members very suited to the task. 
The experts’ satisfaction with information about the audit process, written materials 
and documentation from the institutions varied somewhat, but most considered these 
elements as sufficient. 

According to the experts the panels collaborate as coherent groups. The knowledge 
and competencies represented by each expert is valued - 60% found that their 
knowledge and competencies were utilised effectively by the expert panel. Decisions 
were, according to the experts, reached by consensus and there was little need for 
formal leadership of the panels. Information gathered at site visits, such as statements 
made by representatives of the institution, appear to be of primary importance in panel 
decisions and recommendations, whereas literature on evaluations procedures and 
courses/seminars on auditing were considered as less important in the decision making 
process. 

Credibility of the audit 

The findings of the audits were considered credible by the experts. The experts 
found that the audit process runs a thorough control of the system, the process is not 
dependent on the composition of the expert panels and that audits add knowledge 
and insight about the institutions. The data showed that the appointment as external 
expert had a high status among academics and that serving as an external expert 
was considered as useful experience by higher education institutions. However, 
these observations were somewhat counteracted by the opinion that most colleagues 
appeared to be indifferent to the work of the quality assurance agency and that 
institutions rarely encouraged their academics to participate in expert panels. 

Perhaps the most important aspect of credibility is related to the high degree of 
consensus among the experts on the various questions in the survey. This indicates that 
experts from very different institutions and with varied backgrounds hold the same 
standards and approaches in their work as auditors. They did not experience that their 
professional judgment was restricted by for instance the way the audit process is carried 
out or by the criteria for the audit. 

5.6 Discussion 

These results indicate a high degree of satisfaction. The experts regard their 
competence and knowledge well suited to the task, they perceive commonality in 
views and procedures in the panel’s decision process and see audits as credible and well 
regarded among academics. 

These findings are in part supported by an evaluation of NOKUT carried out by an 
international panel in 2007. The international evaluation by NIFU STEP 23 states that 
the breadth and depth of the expert panel competences received high scores among 
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their respondents 24 . The expert panels were also to a large extent considered “qualified, 
independent and fair ” 25 supporting the experts’ own evaluation of credibility. 

However, a notable part of the respondents (-20%) indicated dissatisfaction with 
the audit processes and results. The international evaluators on their part found a “lack 
of consistency” across expert panels, as well as some cases of unfairness and bias 26 . 

Such tendencies need to be counteracted by the experts themselves and by the national 
agency. These observations also provide an inroad to discuss the weaker aspects of the 
quality assurance processes as well as the role of the external experts in them. 

A critique of the external experts for NOKUT has also been formulated by Raaen 27 
and by Haug 28 . Their arguments mainly relate to methodological issues, specifically 
in program evaluations. But as Stensaker 29 and Elkin 30 have shown, the methods used 
in quality evaluations, whatever their aim or focus, appear remarkably similar. It has 
for that reason been termed a ‘general model’. It may be time for the Norwegian, and 
other national agencies to review not only the application of the general model, but the 
general model itself. In such reviews the role of external experts should be considered 
more closely, as the external experts may constitute the strongest as well as the weakest 
link in the audit process chain. 

There is also obvious danger if the audit processes run too smoothly or if, as Purser 31 
describes it, “the entire evaluation process begins to operate more as a well-oiled 
administrative machine, where the key motivations of learning and improvement are in 
danger”. In the further development of the expert role there seems to be a need to study 
the perceptions and practices of the experts more closely. Qualitative studies on the 
experiences and reflections from participating in audits might be a valuable approach 32 . 
It may also be time to reconsider how experts are prepared and trained for their task. 

5.7 Conclusion 

The perceptions of the external experts in quality audits show broad agreement and 
interesting divergences. A closer analysis can contribute to the enhancement of national 
quality systems and assist in how expertise is enacted by quality auditors. Since the data 
was collected by questionnaires, the issues raised might well be pursued by interviews 
to elicit more detail and depth. Most importantly, the experts should have more 
opportunities to discuss their role, and their perceptions should be valued through 
further research as well as through agency structures. Although limited to Norway, 
these findings should be of general interest to the quality assurance community, as 
NOKUT applies an internationally accepted methodology for national systems of 
quality assurance 33 . 
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Chapter 6: Considering publications: 
The advantages and disadvantages of 
publishing accreditation decisions and 
reports 

Karl Dittrich, the Chairman ofNVAO 

6.1 Introduction 

Our society is becoming increasingly transparent. The decrees and decisions of 
governments and government bodies have to be made public and they must be based on 
sound lines of argumentation and reasoning. Citizens, organisations and companies are 
seeking their “rights” and are understandably no longer satisfied with administrative 
bureaucratic decisions based purely on power. If the parties themselves do not look for 
the reasons for and background to decisions, there is always the press that can call on 
public access rules to uncover the truth. Moreover, organisations that protect the rights 
of consumers and citizens are increasingly better informed and equipped to play an 
important role in the publication of degrees and decisions. 

The time in which decisions could be made in back rooms lies in the distant past. 

The time in which decisions were made as simple announcements, with no explanation 
of the content or accountability, is scarcely imaginable. 

As quality assurance agencies, we are all facing this problem and state of affairs. 
Whether we evaluate, audit or accredit, the duty to make well-founded decisions and 
to make these public is one we must all fulfil. This is, however, not simple, because - 
quite rightly - in a civilised society the privacy of individuals and organisations is also 
protected under the law. Therefore, we sometimes have to wrestle with conflicting 
interests. And if the national legislature has not set down clear rules, we ourselves have 
to ensure the necessary transparency and weigh the interests that are involved. 

The questions to be analysed are: How do quality assurance agencies handle the 
publication of their decisions, which considerations play a role in this and what effects 
can the publication of decisions have on the position and procedures of the assessing 
experts? This article simply tries to describe the complexity of this problem as it is 
relevant to our work: the assessment and promotion of the quality of higher education. 

6.2 The practice in Europe 

In order to prepare for the workshop, I investigated the publication policy of 16 
members of ENQA on decisions regarding the assessment of the quality of institutions 
and/or programmes. I made use of the questionnaire that NOKUT sent out to the 
participants of this workshop. Furthermore, I was able to use secondary sources as a 
basis, namely the bilateral comparisons of content that the members of the European 
Consortium for Accreditation (ECA) have drawn up in recent months and years, as well 
as a number of publications. 

Not all the organisations make decisions. They may also conduct evaluations and the 
need for publication is then much less evident. EUA and the quality assurance agency of 



Baden-Wurttemberg (Evalag) do not publish evaluation reports but rather leave this to 
the evaluated institutions themselves. 

The decisions of our organisations seem to be widely published, but in varying forms: 

• In the most comprehensive form, all positive and negative decisions are published, 
including the underlying reports of the panels: this is the case for all Scandinavian 
agencies except for NOKUT; the Accreditation Organisation of the Netherlands 
and Flanders (NVAO) and the agencies that carry out the assessments within 
this system; the British QAA and the organisations from Estonia and Lithuania. 
NOKUT publishes the reports from audits and institutional accreditations and 
also the formal statements to the reports from evaluated institutions. The reports 
from initial programme accreditations are public, but not published. 

• A less comprehensive form is that in which all decisions are published but without 
the underlying panel reports: this is the practice of the Austrian OAR, the Polish 
PKA, the Bulgarian NEAA and the French CTI. 

• An even less comprehensive form is that in which only positive decisions are 
published. There are three variants in this category: decisions together with 
the panel reports (ANECA), decisions together with the panel reports if the 
applicant consents (the Austrian agencies FHR and AQA, and the Swiss OAQ) 
and summaries of positive decisions and panel reports (the German agencies 
ACQUIN, AHPGS, AQAS, FIBAA, ASIIN, ZEVA and the Akkreditierungsrat). 

6.3 Background 

A number of different - sometimes conflicting - arguments play a role in the discussion 
of the way in which the assessments of quality assurance agencies are published and, in 
particular, the scope of the publications. 

a. The importance of transparency 

In a number of countries, the “transparency” of the actions of the government or 
of bodies authorised by the government to make decisions, is literally set down 
in legislation or administrative practices. In particular in Scandinavia, England, 
Belgium and the Netherlands, transparency weighs so heavily in the balance 
that all documents that play a role in the decision-making must be automatically 
and unreservedly published. The party being assessed is aware of this and can 
therefore take it into account. Needless to say, rules also apply in those countries 
where almost total transparency obliges the assessor to act “properly” (rules of 
“good governance”) and the assessed party has all kinds of rights to hear and be 
heard, to object and appeal, but ultimately every decision is made completely 
public. This transparency is naturally to the benefit of the “public”, whether the 
latter is defined in terms of students, labour market, political arena or society in 
its broadest sense. 

b. The importance of accountability 

From the perspective of government responsibility for access to and the funding 
and quality of higher education, it can be expected that there is accountability 
for the assessed quality of programmes. This is something those politically 
responsible, the labour market, students and their parents (as part of society 
as a whole) are entitled to. For example government regulations about the 
publishing of decisions are often supported by arguments about the large financial 
contribution the government makes to the funding of academic universities and 



universities of professional education. The government virtually always derives 
“rights” from this funding, that have to do with the public accountability of higher 
education institutions. Naturally, the degree of autonomy of the higher education 
institutions plays a role in this, and consequently in the relationship between the 
government and the institutions. But even in countries where institutions have 
degree-awarding power, the accountability obligation is, by and large, not subject 
to discussion. 

An interesting and special type of accountability can be present in 
countries with an extremely open education system, like in the Netherlands 
and Austria, where all kinds of private providers may enter the education 
sector. The relationship between these providers and the government is not 
so much expressed in terms of funding, but rather in the awarding of degrees 
and qualifications for which the government desires quality guarantees. In 
that respect, these private providers are obliged to “comply” with government 
regulations. 

c. The importance of consumer protection and information provision 

The publishing of decisions contributes to the information available to consumers 
of higher education and protects them from qualitatively poor or weak providers. 
This objective can be fulfilled in an active or in a passive sense. Fulfilling in 
an active sense entails publishing both positive and negative decisions; if it is 
fulfilled in a passive sense, then consumers must deduce the possible existence 
of a negative decision about the quality of a programme from the absence of a 
positive decision. This demands a certain amount of interpretation, but if people 
are familiar with the regulations of the country it should not pose any problems. 

It is striking that the form in which information is made available to consumers 
of higher education is not very well developed as yet. In a number of systems, 
the commitment and perseverance of prospective students is severely tested. The 
information made public by the organisations is so large in scope that finding 
relevant information becomes extraordinarily complex! The full scale of this 
problem became clear in the ECA Qrossroads [sic] project aimed at realising a 
single European information tool for accreditation decisions. 

d. The importance of the right to privacy and the protection of sensitive 
information 

In several legal systems, mainly those based on the German legal system, a 
great deal of emphasis is placed on the protection of the interests of the assessed 
parties. The basis of this approach is to be found in a dislike of an overly powerful 
state and the protection of the individual (a person as well as an organisation) 
in respect of the state. Not making negative decisions public, for example, is 
based on privacy legislation and breaching, as this in general leads to severe 
penalties. Moreover, account must naturally be taken into consideration regarding 
competition. Institutions and programmes find themselves in a situation where 
there is competition for students and lecturers. From this perspective, it is 
therefore not a matter of course that all information and decisions are made 
public. 

e. The prevention of the improper use of decisions 

In several systems, in addition to the arguments above, another argument is used 
to limit the publication of reports and decisions: the prevention of the improper 



use of this information. This argument is mainly used to discourage ranking. 

Often in journalistic practice, assessments of panels are rated in a particular 
way and then added up to arrive at rankings. This is considered undesirable for a 
number of reasons: firstly, the quality assessments of panels are not by definition 
comparable due to the different composition of the panels; secondly, not all of 
the subjects assessed are of equal significance for the quality of a programme; 
and thirdly, the comments, often aimed at improvement, are taken as absolute 
judgements whereby a distorted image of the quality of a programme is created, 
etc. 

f. The importance of legal tenability 

If the decision is made for the complete publication of decisions, lines of reasoning 
and assessments, people should be prepared for possible debates and legal 
proceedings as a result of unequal assessment, a possible lack of consistency 
between different assessments and for the extremely careful consideration and 
substantiation of assessments. In some systems this would lead to a very heavy 
set of formal requirements for decision-making that could have the consequence 
of highly bureaucratising tendencies. In practice, this could result in sizeable, 
difficult to read and relatively veiled reports. 

g. The protection of panels and their members 

As a final consideration, it can be pointed out that in practice, although in a 
number of systems decisions are published, the underlying panel reports are 
not made public. This pertains mainly to those systems in which recourse to 
the courts is very easy. In some systems, it happens that the panels as a whole 
(or even individual panel members, particularly the chair) are threatened with 
legal proceedings by individuals or institutions. This is, of course, extremely 
undesirable within the framework of peer reviews and it is understandable that 
panels are protected against such practices. Legal proceedings should then be 
instigated against the quality assurance agency rather than the panels. 

Those who are responsible for the policy regarding making the results and the 
underlying panel reports public therefore have to take into account a large number of 
different interests. Some of these interests are in opposition to one another, as is the 
case for the complete publication in favour, for example, of a rational decision-making 
by students about their choice of study versus the right to privacy of the programme, 
the institution and individuals. The considerations made and to be made are often 
expressions of the political culture in the country concerned. These cultures have great 
differences, varying from total and virtually unrestricted transparency to forms of 
transparency that take individual interests into account relatively strongly. 

6.4 Panels and publications 

External experts are crucial to assess the quality of education and research. Panels 
must therefore be compiled with the utmost care because the independence, expert 
knowledge and authority of the panel are the deciding factors for the legitimacy of the 
assessment of a programme or institution. This legitimacy is, of course, also necessary 
for the outside world to be able to trust the assessment of the experts. But this 
legitimacy is even more important for the party being assessed: people must know they 
are being treated fairly, they have to be able to trust the assessment of the experts and, 



moreover, have the feeling that the implicit or explicit recommendations set down in 
the panel report are to the point. 

Certainly when quality assurance agencies have to make momentous decisions 
about the existing or absent quality of a programme or institution, the legitimacy of 
the panels must be above reproach. This applies even more strongly if sanctions are 
imposed on the basis of the decisions of quality assurance agencies, sanctions that may 
have consequences for funding, the right to award degrees, or the enrolment of new 
students. 

The panels already bear what is in itself a heavy responsibility. If, on top of this, the 
reports of the panels were to be published, this would demand the careful supervision 
and training of the panels and their secretaries. In general, the panel reports are 
sufficiently diligent and balanced. The panels are aware that they must comply with 
criteria, and in addition they are usually quite capable of finding the right balance 
between assessing the basic quality to be evaluated and making recommendations for 
improving the quality of the programmes and institutions. Diligence and the balance 
between assessments and recommendations ensure the legitimacy of the panels and the 
large majority of our assessment procedures do not seem to present any problems in this 
respect. However, as a warning, the publication of the reports may adversely influence 
the behaviour of the panels by causing possibly one of the following effects: 

a. The “macho” panel 

It is imaginable that panels will adopt an extra forceful attitude to show that 
they “dare”. For example, in the Dutch context we have experienced that 
extremely authoritative professors now desire to be involved as panel members 
in the accreditation system, while they were unwilling to do so under the 
“softer” assessment system. The reason given was literally that the fact that 
their assessments could have an impact would make their efforts worthwhile. 

In their eyes, the former assessment system was too non-committal and this 
repelled this very powerful and authoritative group more than the possible public 
consequences of their assessments in the accreditation system. The game has to 
be played well and fairly. In Flanders, the chair of a panel was recently threatened 
with legal proceedings by an assessed professor after the panel had made such 
critical comments about the weight of a particular subject in the programme, 
that the programme - in accordance with the Flemish basic rules regarding 
accreditation applications - could not be accredited; the situation became even 
more threatening when the institution involved also appeared to be instigating 
legal proceedings against the panel. If that would have happened, the legitimacy 
of a system with peer reviews would be gone. However, in my view, the risk 
that a much more procedurally oriented assessment system will be imposed 
seems greater than the risk that a government would be satisfied with a “softer” 
evaluation system. 

b. The “old school tie” panel 

A very different effect of the publication of their reports occurs in panels that 
allow their own role during an assessment to evolve from that of assessor into that 
of advisor. Such struggles can often be read in panel reports: the arguments in 
favour of considering an assessment “satisfactory” are not supported by the facts 
observed by the panel but rather by the willingness of the assessed colleagues to 
make amendments, implement or change policy, or to improve behaviour. The 



assessment of an actual situation is then converted into an “assessment of a plan”, 
which is, of course, something else entirely. This type of procedure can often be 
tracked back to the personalities of the panel members: there is mutual respect 
and in the worst cases it is an example of the non-intervention principle: “this 
panel will not be too hard on you if you won’t be too hard on us later”. The formal, 
proper independence of a panel cannot resolve this type of behaviour. However, it 
could often help if true outsiders were added to the panels, especially if they were 
to come from abroad. But even then, such behaviour cannot always be prevented, 
certainly not in the smaller disciplines or specialisations, 
c. Veiled language 

In assessment practice, you can often see that during site visits, panels are more 
forceful, but have a softer approach later in writing. In itself this is understandable 
because the written report has to provide careful assessments and lines of 
argument. This often involves some caution because an assessment is seldom 
black or white or yes or no. Usually, assessments are shades of grey and demand 
the weighing of various views before a judgement can be made. 

NVAO was confronted with this in the earliest phase of its existence when it 
chose to be highly pro-active in the assessment of the panel reports. Negative 
comments and statements became blown out of proportion as it were, whereas 
a positive assessment brought the panel into doubt. This had in part to do with a 
learning curve. In an accreditation system, the assessments of a panel obviously 
have to be stronger as well as better substantiated than in a system aimed 
at improvements. As the body is ultimately responsible for the assessments, 

NVAO therefore had to adopt an accurate and exact position. This position, 
however, appeared to have the reverse effect on the quality of the reports 
because panels became reluctant to include negative comments in their reports. 

It took a long time before the panels dared to do so again. Only when NVAO 
had publicly admitted it had gone too far in filtering the reports did the panels 
have enough confidence to again base their public considerations in the reports 
on positive as well as less positive findings. Incidentally, in the Netherlands, 
there are still institutions that ask panels to send their recommendations as 
non-public management letters to avoid a situation in which NVAO would not 
follow a positive assessment of a panel or to prevent competitor institutions and 
programmes from receiving too much information. 

6.5 Conclusion 

Publishing information regarding the quality of institutions and programmes is not 
a matter of course and there are different perspectives from which the importance of 
publication can be viewed. Moreover, publishing can have an effect on the procedures 
of the panels. It is therefore understandable that a lot of thought is being given to 
the advantages and disadvantages, and the possible positive and negative effects of 
publication. Furthermore, the political culture and particularly the prevailing views 
about the position of the state play an important role in the shaping of a broad or 
reserved publication policy. 

From a different viewpoint, ambitious institutions have thought about and continue 
to think about their mission; they base their vision and goals on this mission, and 
continue to develop it further. Such institutions offer something to both society and 



students: a curriculum with a particular content that may also have been developed 
on the basis of an educational model. Such institutions promise something to the 
labour market and to their students on the basis of which they try to attract students 
and the interest of the labour market. They are actively engaged in marketing and 
public relations. Quality assurance agencies often assess whether an institution or 
a programme delivers on its promises at an internationally acceptable level for a 
bachelor’s or master’s degree programme. Promises must be kept: if an institution or a 
programme is failing to deliver on its promises, this should be made known. And this is 
an argument as great as possible transparency in decisions. 

From the perspective of the European Higher Education Area, most European 
countries, education institutions, students and a large proportion of the economically 
most important branches of industry - all benefit from high student mobility, expressed 
in a considerable amount of foreign experience. If this viewpoint is shared, it is 
evident that sufficient and reliable information should be available about the quality 
of institutions and programmes. Ideally, this information would be provided by the 
institutions themselves. But precisely for the sake of objectivity and to prevent one- 
sided marketing and public relations, I believe that the assessments of quality assurance 
agencies constitute an additional guarantee. 

As a product of my own political system I consistently stress the benefits of the 
publication of independent assessments more strongly than the disadvantages. As a 
former chair of a university (Maastricht University) that has chosen an educational 
approach that is entirely its own (Problem-Based Learning) and that has, moreover 
with full conviction, chosen strong internationalisation, I have always been a supporter 
of as much transparency as possible regarding quality assessments. This position 
obliged my university to be constantly alert to possible drops in quality and to always 
strive to deliver on our promises. The publication of decisions and the reports of the 
expert panels will ultimately have a positive effect on the quality of institutions and 
programmes. 



Chapter 7: How to promote the 
usefulness and the utilisation of 
evaluations 



Project Manager Sirpa Moitus, FINHEEC 
Chair, Professor Riitta Pyykkd, FINHEEC 

The aim of this article is to present the practical procedures that the Finnish Higher 
Education Evaluation Council, FINHEEC, has used to ensure the quality and 
effectiveness of evaluation data. Some of the procedures are specific to Finland, some 
seem, on the basis of the survey conducted by NOKUT (the Norwegian Agency for 
Quality Assurance in Education), to be in use in other countries as well. 

Since the start of its operations in 1996, the Finnish Higher Education Evaluation 
Council has based its evaluations on the developmental-oriented approach. The 
aim of a developmental-oriented evaluation is to produce information and to carry 
out evaluations in such a way that serves the higher education institutions in their 
development work 34 . 

This article has been structured in accordance with the phases of an evaluation 
process. The article contains experiences relating to different types of FINHEEC 
evaluations (institutional evaluations, programme and thematic evaluations, audits, 
selection of centres of excellence in teaching and learning). The article is based on the 
experiences of the FINHEEC Secretariat, surveys on the utilisation of evaluation data 
and on feedback regularly collected from higher education institutions. 

7.1 Selection and induction of experts 

One of the crucial ways to ensure the usefulness of evaluation data is related to the 
selection of expert teams and their training. 

The recruitment criteria for the evaluation and audit teams are transparent, and 
the evaluators are selected by decision of the FINHEEC Council, which is public. The 
most usual selection criteria for evaluators include knowledge of the sector or theme 
to be evaluated and of Finnish education and evaluation. Participation in long-term 
evaluation training, or practical evaluation experience acquired in internal or external 
evaluation projects of higher education institutions, is considered an advantage. In the 
audit teams, previous evaluation experience is necessary, and an attempt is also made 
to always recruit a few people who have previously acted as auditors in audit teams. 

As a rule, Finnish and other Nordic evaluation teams include a student 
representative. In Finland, stakeholders and employers are also represented in the 
evaluation teams. In these instances, it might be better to use the term external 
evaluation team instead of peer evaluation. 



34 The methods of developmental-oriented evaluation and the experiences gained have been reported in the report: Moitus, S. & 
Saari, S. 2004. From Methods to Development. The Evaluation Methods of FINHEEC 1996-2003. FINHEEC Publications 
10:2004. (The language of the publication is Finnish. Abstract is available in English.) 



The participation of students in the evaluation teams is important not only in 
ensuring the comprehensiveness of the student perspective, but also in disseminating 
the results. When students participate in evaluation visits as equal team members, they 
have greater potential to further circulate the effectiveness of the evaluation results 
through the student unions. 

The audit teams receive a compulsory two-day training which focuses on the 
objectives, methods and criteria of the audit process. A lot of attention is paid to 
the practical implementation of the visit, for instance to audit techniques, question 
phrasing and reporting. Former auditors are invited to the training sessions to share 
their experiences. Additionally, institutional quality managers have been invited to 
introduce the quality assurance system and related web documentation to the auditors 
prior to the audit visit. According to the feedback, these have been the most interesting 
sections to the trainees. 

7.2 Evaluation process 

The method of organising the evaluation process has a major impact on how useful the 
higher education institutions experience the evaluation. 

In the starting phase of the evaluations, FINHEEC aims to ensure the usefulness of 
the evaluation data in the following ways: 

• The higher education institutions may propose fields or themes to be evaluated; 

• The higher education institutions have a role in the planning of the evaluations; 

• Evaluations normally start with a launching seminar which offers the institutions 
a forum to comment on the aims and implementation of projects. 

One of the characteristic features of the FINHEEC developmental-oriented evaluation 
is the tailoring of the evaluation methods. In practical terms, this may mean e.g. 
replacing or supplementing the self-evaluation phase or institutional visit with some 
other method. The aim is to find appropriate methods that best fit each evaluation 
target and at the same time, that encourage wide participation and joint developmental 
effort among staff and students. In this way, the institutions are expected to better 
commit towards the evaluation results. 

In some thematic evaluations, self-evaluations have been carried out in pairs as a 
process between the staff and students, or in pairs between degree programmes within 
an institution. In the evaluation of university teacher training, the Delphi method 35 was 
used to outline the future of teacher training, as well as skills and knowledge required 
from future teachers and, also, a portfolio technique was used to complement the 
normal self-evaluation method. 36 

In 2004, FINHEEC conducted a survey on the usefulness of evaluations 37 . According 
to the degree programmes that had participated in FINHEEC evaluations, the strength 
of the self-evaluation was that it offers an opportunity to critically examine teaching 
and learning and other activities in a structured framework. A selective attitude and 
“embellishment” of issues were seen as threats and weaknesses in the self-evaluation 



35 The Delphi method is an iterative process for consensus-building among a panel of experts. The method is used for forecasting. 

36 A portfolio is a collection of works by an individual or a group. For the evaluation of teacher education, the process of gathering 
the portfolio was important, as it involved 1) gathering evaluation material through structured questions 2) reflection and 
selection by the evaluated institution of material to be included and 3) a self-evaluation, where the chosen material was presented, 
analysed and the past and future evolution of teacher education was demonstrated. 

37 Moitus, S. & Seppala, H. Mita hyotya arvioinneista? [What is the use of evaluations?] Report on the use of evaluations in the 
education sector carried out by the Finnish Higher Education Evaluation Council between 1997 and2003. FINHEEC publications 
9:2004. 



procedure. Especially, if external incentives are linked to the evaluation, such as the 
allocation of funds for a centre of excellence, the report may resemble an application, 
which presents the institution’s strengths. However, in the selection of centres 
of excellence in teaching as well as regionally, there is seldom a link between the 
evaluation results and resource allocation. Normally, in the FINHEEC evaluations, the 
evaluations do not have direct financial consequences. 

The tailoring of evaluation methods has played a significant role in thematic 
evaluations. Thematic evaluations are by their nature cross-evaluations, i.e. they 
encompass all the units of the university and/or polytechnic sector or several of their 
units. Examples of FINHEEC’s thematic evaluations include the Evaluation of the Open 
University, the Evaluation of Student Guidance Services and Evaluation of Student 
Admissions in Universities. It has often been necessary to start these evaluations with 
a significant background assessment or with a survey, and in this way new, national 
summary data has been made available. 

In thematic evaluations, a national discussion forum has often been organised to 
supplement or replace the self-evaluation stage, to which representatives of social 
stakeholders have been invited in addition to education providers. These discussion 
fora have been given various names: hearings, thematic discussions or round-table 
discussions. The participants of these events have consisted of education providers and 
stakeholder representatives who have been invited to reflect the current situation and 
future challenges of evaluated theme. 

From the perspective of the evaluation teams, the advantage of the discussion 
fora is that they have offered the evaluation teams the opportunity to test their own 
conclusions and their relevance in a public forum before the report is drawn up. 

The implementation of audits of an institutional quality assurance system has a 
specified form. It is based on coherent audit targets and criteria defined in the Audit 
Manual 38 . A developmental-oriented evaluation approach can, however, also be applied 
in organising the audit visit, for instance. In a few audit projects, representatives of two 
faculties or degree programmes have been interviewed together simultaneously, so that 
these representatives have been able to compare and benchmark each other’s quality 
assurance practices. At the same time, the audit team has gained an understanding 
of the similarities and differences in quality assurance within the higher education 
institution. The audit teams have also organised thematic interviews whereby 
understanding can be gained in one interview. The gained understanding can relate 
to, for example, the pedagogical development or staff recruitment and development 
at the various levels of the higher education institution, namely in the administration, 
faculties, departments and degree programmes. 

7.3 Reports 

All the FINHEEC evaluation reports are made public. They are published either in print 
or electronic format. The Council discusses the policies on writing the reports and the 
Secretary General, as the editor-in-chief of FINHEEC’s publication series, reads all 
the reports and ensures a common line. The evaluation reports aim to give constructive 
criticism and to produce concrete information or proposals that assist the institutions 
to genuinely develop their activities. The reports also highlight the strengths of the 
operations and good practices. 



38 FINHEEC Audit Manual for 2008-2011 is available at www.kka.fi/pdf/julkaisut/KKA_1007.pdf 



In programme, and especially in thematic evaluations, FINHEEC has given the audit 
teams a fairly free hand in preparing the report, which has been seen in the variety in 
terms of content and scope of the reports. On the other hand, the audit reports follow 
a common structure which compiles in the final chapter the strengths, good practices 
and development targets of the institutional quality assurance system. 

The uniformity of the audit reports is especially important because the Council 
decides based on the audit reports whether the higher education institution either 
passes the audit or if its quality assurance system has major shortcomings and thus 
requires a re-audit within two years. 

In accordance with the principle of autonomy, each higher education institution has 
been able to choose a quality assurance system suitable for their institution (e.g. based 
on the ISO standards, EFQM, BSC, CAF or self-developed QA system). The task of the 
audit team is to evaluate the comprehensiveness, transparency and effectiveness of the 
institutional quality assurance system in relation to its strategic objectives. One of the 
challenging tasks of the audit teams is therefore to balance the coherence of the reports 
on the one hand and the institution specific feedback on the other. 

7.4 Publication event: a means of disseminating information on the evaluation 

One of the methods specific to FINHEEC is to arrange a publication seminar on the 
evaluation report. These seminars are open to the entire institutional community. A 
typical programme includes the introduction of the evaluation results by the chairman 
and comments by the representatives of the institutional management, staff and 
student representatives. 

Such concluding seminars are a means of increasing the publicity and 
effectiveness of the reports, but they also offer an opportunity to prevent possible 
misunderstandings. In addition, they are a communal way of discussing evaluation 
data. The publication seminars have proven to be very important in instances when the 
higher education institution has received a critical result in the evaluation, such as a 
re-audit decision. On some occasions, the rector of the institution has used the forum 
to present his or her own programme of measures to rectify the shortcomings presented 
in the report. 

7.5 Handling evaluation data within higher education institutions 

The way in which the evaluation report is handled within the institution after the 
publication of the report also has an impact on the utilisation of the evaluation data. 

According to the feedback received to date, the most common way higher education 
institutions handle the evaluation reports is to discuss the recommendations and 
strengths in conjunction with staff planning and development days. The FINHEEC 
surveys showed that after the degree programme evaluations, several degree 
programmes and subjects had organised information and discussion events addressed 
to the staff and students to promote the utilisation of the evaluation results. FINHEEC 
cannot, of course, demand that the institutions process the reports in a specific way, 
but it can bring matters to the institution’s attention by asking for further consideration 
of the report in conjunction with the feedback surveys. 



7.6 Follow-up evaluations 

The principles of the Finnish Higher Education Evaluation Council include the 
implementation of follow-up evaluations around three years after the initial 
evaluation. Follow-up evaluations have been used to establish whether development 
recommendations have been implemented. The follow-ups have been considered as a 
means of furthering the effectiveness of the evaluations. For example, according to the 
follow-up evaluation of education in social work and social services, the evaluations 
have had a significant impact on increasing the connection between teaching and 
research, the development of curricula and networking with the employment sector. 

According to the higher education institutions, the general strength of the follow-up 
evaluations is the fact that they ensure the implementation of the evaluation results and 
force the unit to re-evaluate its own operations. It must, however, be stated that the 
institutions are not always able to specify which changes are a result of the evaluation 
and which result from changes to the operating environment or from the Bologna 
Process, for instance. 

The above-mentioned FINHEEC survey produced similar results. At faculty and 
programme level, the evaluations have had the biggest impact on the development of 
curricula and teaching content, as well as on the development of student guidance 
and counselling. The evaluations have also resulted in more attention being paid to 
establishing an international focus. Additionally, evaluations have produced new ideas 
for the development of internal evaluation and feedback systems. However, not all the 
development proposals suggested by the evaluation teams were implemented in degree 
programmes. This was mainly justified by the lack of financial and time resources. 

It is interesting that when the same question on the use of the evaluations was asked 
to the rectors, they felt that the evaluations had also had a strategic effect. The rectors 
believed that the evaluations of the education sector had (1) supported the evaluation of 
the institutions’ own operations, (2) resulted in concrete development actions and (3) 
created an overall picture of the national state of the sectors being evaluated. 

The results of the evaluations were also utilised in the central administration of 
the institutions, which could not be interpreted from in the answers from the degree 
programmes and subjects. The higher education institutions valued not only the 
unit-specific feedback but also the national reference data on their own sector that 
was elicited from the evaluations. It remains for the evaluation agency to consider how 
the evaluations of the education sector can best serve both perspectives - both unit- 
specific and national development. 

It can be stated, on the basis of the feedback collected by FINHEEC and the audit 
reports, that the audit procedure has sharpened up the systematic development of 
quality assurance systems and the documentation of quality assurance practices. The 
audits have directed higher education institutions to view their operations from an 
overall perspective and seem to have contributed to clarifying the connection between 
quality management and institutional management. The higher education institutions 
have also increased quality training for their staff and students. 

7.7 Future challenges 

There are several future challenges relating to the utilisation of evaluation data, two of 
which are considered below. 



DISSEMINATING GOOD PRACTICES 

Evaluation reports produce a great deal of information on good practices relating to 
the institutions’ operations. However, the challenge is how to disseminate these good 
practices as effectively as possible, so they can be used by other higher education 
institutions. 

ANALYSES AND RESEARCH ON THE EVALUATIONS 

The FINHEEC Council and the higher education institutions have a great need to 
obtain analyses and research on the evaluations. Although the Secretariat prepares the 
summaries of the evaluations, long-term and external research work is also needed 
on the use of the evaluation data. In view of this, FINHEEC has laid down guidelines 
in the new Action Plan for 2008-2009 stating that the agency could allocate funding 
for research on evaluations, which could therefore generate more research data on the 
evaluations. 



Chapter 8: Reasonable expectations 
and ambitions for external evaluations 



Gro Hanne Aas, Wenche Froestad 

There are varied opinions on the actual value of evaluations carried out by quality 
assurance agencies. In the critique it is argued that evaluations do not address 
educational quality adequately. Another complaint is that the evaluation reports result 
in unfounded conclusions about the quality of the institution or programme when 
presented in public. To better understand what evaluations are, what can be accomplish 
with them and what they can be used for, a more fruitful approach would be to ask: 

• What can be considered as reasonable expectations towards these evaluations 
from the evaluated institutions and from the society? 

• What can be considered as reasonable ambitions from the side of agencies and 
experts? 

8.1 Evaluations in a new regime: Changes in the status of expert panels' 
knowledge production? 

Quality assessments by external expert panels play a crucial role in state governance 
of higher education in many countries (see Askling in this report). In the Norwegian 
context, this has been the case for a while. Does the status of expert knowledge change 
with the implementation of a national system where evaluations are directly linked to 
sanctioning and rewarding? 

The formation of the independent Norwegian Agency for Quality Assurance in 
Education, NOKUT, in 2003, represented a change in the steering model of higher 
education. The purpose of NOKUT is to oversee the quality of Norwegian higher 
education by means of evaluation, accreditation and approval of quality systems, 
institutions and study programmes. A basic element in the model is that the institutions 
of higher education are fully responsible for both the quality of their provision and 
the quality assurance of their educational activities. There is nothing new in this 
responsibility as such, but the institutions are now required to demonstrate how 
responsibility for quality is followed by actual quality assurance. It is mandatory for all 
institutions to have a quality assurance system that covers all aspects of educational 
provision. When a quality assurance system is evaluated, emphasis is placed on whether 
an annual report is presented to the board of the institution offering a coherent and 
overall assessment of educational quality as well as an overview of plans and measures 
for continued enhancement work. A crucial part of the model is that accredited 
institutions are granted rights of self-accreditation according to their institutional 
status. 39 

There are three new elements in the Norwegian system: 

• the general possibility for institutions of higher education to apply for a change in 
their institutional status through accreditation 

• NOKUT’s right to revise the given accreditation of study programmes or higher 
education institutions with a possible sanction of withdrawing accreditation 

39 Universities may grant degrees on all levels, specialised university colleges may grant degrees on all levels within their academic 

fields and university colleges may grant degrees for bachelor programmes. 



• NOKUT’s duty to conduct cyclical audits of the institutions’ QA-systems and 
to approve or disapprove these, with the possible sanction for the evaluated 
institution of losing its self-accrediting rights or the right to apply for 
accreditation of a new educational provision. 

In the Norwegian system, external control of new educational provisions is not a new 
phenomenon. 4 °However, with NOKUT there has been a marked increase in the use 
of evaluations conducted by external expert panels, and hence, an increased amount 
of reports presenting knowledge about higher education institutions. The evaluations 
are carried out as parts of a national system for quality assurance with reward and 
sanctioning at the core. NOKUT’s Board of Governors functions as the decision- 
making body for approval or non-approval. The legitimacy of this decision-making 
rests heavily on the quality and adequacy of the expert panels’ reports, and on the 
reputation of the experts themselves, as well as on the roles ascribed to them. 

8.2 Reasonable ambitions: What does the QA-agency need to know? 

With the institutions of higher education themselves in the centre of attention for 
evaluations, one may argue that the evaluations should be carried out for the benefit 
of the institutions. In fact, the Norwegian act relating to universities and university 
colleges states: “Accreditation and evaluation activities shall be designed in such a way 
that the institutions can benefit from them in the course of their quality assurance and 
development work”. Accordingly, the evaluations are tailored to fulfil this requirement. 
Still the evaluation report from the external expert panel is the main basis for the 
Board of Governors in NOKUT to make a decision of approval or non-approval. 

8.2.1 Accreditation: evaluating fulfilment of standards and criteria 

When the purpose of an evaluation is the decision on accreditation or revision of 
accreditation (of a programme or an institution), it is vital to know whether the 
predefined minimum standards of quality are fulfilled. In a society where there are 
several providers of higher education, this argument is both legitimate and important. 
However, one must not forget the limitations of purpose and scope of this type of 
evaluation. The purpose is to provide necessary and adequate information about the 
quality of a study program or a higher education institution, according to the defined 
requirements in order to make a correct decision of accreditation. Such assessments are 
not designed to cover all aspects concerning the quality of a program or an institution 
under review. 

The legitimacy of an accreditation process depends heavily on whether the 
predefined standards are understood and accepted by the evaluated institutions and 
by the experts conducting the evaluation. The expert panels’ major task is to find out 
whether, or to what extent, the programme or the institution under evaluation can 
meet the pre-set standards. The panels are not allowed to bring about new standards 
in addition to those predefined. It also is a necessity that the standards are applied 
consistently from one evaluation to another. If these conditions are fulfilled, the 
accreditation decisions can maintain a high level of legitimacy among all stakeholders 
and in the society at large. 



40 Earlier, universities applied for new master programmes and Ph.D. programmes, state-owned university-colleges apply for study 
courses lasting more than one and a half years, and private-owned university colleges applied for any new study course. 



Accountability or control is often considered to be a main focus in accreditation 
assessments. However, a panel’s task will not only be to conclude whether standards are 
met but also to point out directions for further development. Thus, the enhancement 
perspective should not be underestimated, but it is still related to the pre-defined 
standards. 

8.2.2 AUDIT: EVALUATING AUDITABLE PERLORMANCE 

Like accreditation, audit also has a certain scope of evaluation and a limited purpose. 

In our context, audit is a (meta) evaluation of the functioning of internal systems for 
quality assurance that the institutions are required to have . 41 The focus of an audit 
is an evaluation of a planned and systematic quality work. The basis for the external 
expert panel’s evaluation is the institution’s own documentation of its (auditable) 
performance 42 , the institutions’ own ability to evaluate the quality of its provision, and 
the ability to act upon this information in order to obtain continuous improvement. 
Unlike accreditations, the criteria for evaluations in the Norwegian quality audits are 
not formulated as standards to be fulfilled. The criteria are more of a generic type 
and include the European Standards and Guidelines for internal quality assurance in 
institutions of higher education, adopted through the Bologna process. The evaluations 
cover an institution’s quality assurance system as a formal and participatory structure. 
By producing documentation and by assessing the institution’s quality throughout its 
portfolio, the institution is able to set priorities and take measures in order to obtain 
improvement. 

In practice, the expert panel first reads the documentation requested from the 
institution. During their visit to the institution, the panel interviews, in addition to the 
leadership, a selection of teachers, students and administration staff, as well as some 
external stakeholders 43 to a certain degree. The interviews provide the panel with an 
impression of the implementation and the use of the system. They provide information 
from experienced actors and thus offer a broader view on the quality work at the 
institution in question. Informal information put forward by ad hoc actors, media and 
others should not be taken into consideration by the panel. 

However, in an audit, the bulk of information presented to the panel has usually 
already been constructed in a specific form in order to support the quality assurance 
system. This notion will further mark the conversations during a site visit. Thus, it is 
crucial to explore what the strong focus on auditable performance can mean to the 
other forms of quality work in an educational environment. In order to understand the 
value as well as the limitations of an audit as a form of evaluation, it is necessary to 
reflect upon the implication that audits actually deal with auditable performance. 

8.2.3 KNOWLEDGE AS ORDERED? 

In Norway, the standards for institutional accreditations are laid out in the regulations 
to The Universities and Colleges Act of 2005 act set by the Ministry of Education and 
Research. Standards for accreditation of study programs and evaluation criteria are 
specified by separate regulations and documents conducted by NOKUT. Hence to 



41 For a discussion on the relationship between internal and external quality assurance in the Norwegian system, see Gro Hanne Aas 
and Jon Haakstad: “Quality Begins at Home. External Audits as ‘Special Events’ in Internal Quality Assurance Processes”, in EUA 
Bologna Handbook. Making Bologna Work, March 2007, B 4.6-1, Raabe. 

42 On “auditable performance”, see Michael Power, The Audit Society. Rituals of Verification. Oxford University Press, 1997. 

43 Normally these include one or two external members of the Board of the institution and representatives from the field of practise 
in study programmes where practical training pays an important part of the students’ learning. 



clarify the standards is by large a responsibility of the Ministry of Education and 
Research as well as of NOKUT. The standards and criteria have been on a hearing in 
the sector of higher education and among stakeholders, and some adjustments have 
been made according to the received suggestions. Prior to the decision on a national 
quality assurance model, a pilot project was carried out by NOKUT’s predecessor in 
quality assurance. In a report from this project, the main aspects of educational quality 
are discussed . 44 However, the standards and criteria cannot be claimed to have been 
developed on the basis of the research on what measures constitute and stimulate 
educational quality. 

The discussion of the limited scope or purpose of the assessments should not be 
interpreted as away of diminishing the value of external quality assurance. At the same 
time, as pre-set standards and criteria standardise evaluations, they also represent 
the intentions of transparency, equity and fairness in a national system. However, it is 
important that both the agencies and expert panel members reflect on the boundaries 
of the knowledge produced in the various sorts of evaluations. Such reflection will 
promote the formulation of reasonable ambitions for various evaluations. 

8.3 Reasonable expectations: The evaluations in public 

An evaluation report is the formal end product of an external panels’ knowledge 
production. In the Norwegian accreditations and audit reports, the panels sum up their 
findings in an argued conclusion of whether they advice NOKUT’s Board of Governors 
to approve or disapprove. The panels also have within their mandates to point to 
areas for further development. As no additional information is given to the evaluated 
institutions from the evaluations, the reports must contain all relevant considerations 
by the expert panel. 

In audits, institutional accreditations and revisions of accreditations, the evaluated 
institution is invited to give a formal statement to the report. The statement is included 
in the report on the NOKUT’s website. Through their statements, the institutions 
may revise the public picture given in the reports, and influence the decision made by 
NOKUT’s Board of Governors. 

In addition to possible formal sanctions, the evaluation may be a matter of prestige 
to the evaluated institution. The expert panel’s conclusions are often highlighted in 
the regional media while statements from the institutions may be ignored. Several of 
the evaluated institutions have experienced the problem of the reports being conceived 
as “the whole and unquestionable truth”. As discussed above, evaluation reports 
can certainly bring about important information concerning a study program or an 
institution, but they will never represent “the whole truth”. 

For a better public understanding of the reports of expert panel evaluations, the 
focus needs to be on reasonable expectations and adequate ambitions on the evaluation 
itself. This is vital in order to avoid making unfounded claims about the knowledge they 
represent, and on the other hand, to recognise what type of knowledge is produced in 
the evaluations. 



44 See: Peder Gravem, Berit Askling, Margrete Fuglem, Svein O. Haaland, Karianne Jonson, Ingvild Knudsen and Jon Naustdalslid , 
Network Norway Council publication 1998: “Basert pa det fremste...” (In Norwegian) 



Annex I: ENQA workshop on 
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Organised in cooperation with the Norwegian Agency for Quality Assurance in Education 
(NOKUT) 

Oslo, Norway, 14-15 February 2008 
14th February 

09:00 Welcome and opening of the workshop 

Chair for opening and session 1: Gemma Rauret, director of National Agency 
for Quality Assessment and Accreditation of Spain (ANECA) and ENQA Board 
member 

Head of Quality Audits Unit, Jon Haakstad, NOKUT 

Session 1: External expert panels' evaluation vs. research based 
evaluations of higher education 

Professor Peder Haug, Volda University College, Norway 

10:30 Session 2: The quality contract: experts and agency rules - 

how to balance expertise and regulated procedures and standards 

Chair for session 2: Patricia Georgieva, National Evaluation and Accreditation 
Agency (NEAA) and ENQA board member 

Agency regulations meet expert panels - introductory panel 

Karen Junker, external expert in NOKUT, director of education at County 

Governor in Aust-Agder, Norway 

Kirsten Hofgaard Lycke, external expert in NOKUT, professor at University 
of Oslo 

Christian Thune, former president of ENQA 
Panel coordinator: Wenche Froestad, NOKUT 

12.30 European Consortium for Accreditation’s (EC A) code for selection of review 
experts 

Rolf Heusser, director of Center for Accreditation and Quality Assurance of 
the Swiss Universities (OAQ) 

13:00 Performing the quality contract 

The participants of the workshop were, prior to the workshop, asked to answer 
a few questions on the tasks and status of experts in their own agencies. 
Presentation of the summary of results: Advisor Anne Karine Sorskar, 




(NOKUT). 



14:00 Small-group discussions on the practices of the participants’ own agencies 
See separate information in the participants list. 

15:30 - Plenary discussion on session 2 
17:00 

19:00 Dinner at Ekebergrestauranten 



15th February 

09:00 Session 3: Expert knowledge in public 

Chair for session 3: Teemu Suominen, ENQA 

The variety in European practices on publicity of review reports 
Presentation by Karl Dittrich, chairman of Nederlands-Vlaamse 
Accreditatieorganisatie (NVAO) 

10:00 Expert knowledge in public - discussion by panel of Norwegian HEI 
representatives 

Ernst Hakon Jahr, former rector, University of Agder 
Torunn Klemp, rector, Sor-Trondelag University College 
Ingrid Torsteinson, head of Nursing Department, Bergen Deaconess 
University College 

Panel coordinator: Gro Hanne Aas, NOKUT 

11.30 Plenary discussion of session 3 

12:30 Closing of the workshop 

Summing up: Gemma Rauret 
Gro Hanne Aas (NOKUT) 



Closing: Teemu Suominen 




This report is a product of an ENQA Workshop “Assessing educational quality: Knowledge production and the role of experts” 
hosted by the Norwegian Agency for Quality Assurance in Education (NOKUT) in Oslo in February 2008. The workshop gathered 
representatives from higher education institutions and agencies to exchange and discuss experiences on the process of external 
reviews and on the role of the review experts in that process. Also the ENQA pool of experts was invited to this event. The 
workshop and this publication flourished from the “Expert knowledge” project carried out by NOKUT, which focused on the 
status and legitimacy of the knowledge acquired through external assessments of higher education. In external assessments, the 
role of external reviewers (experts) as well as the cooperation between experts and quality assurance agencies are crucial. This 
report aims to address the debate further on the conditions of legitimacy of evaluation results, the role of experts and the agencies’ 
procedures, as well as on the responsibilities of producing publicly available information about the quality of higher education. 
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